phpBB question ....

[extropalopakettle]

I'm not familiar with phpBB, but I'm a fairly competent and adept developer, and have an idea for a modification or plugin I'd like to write, almost kind of as a joke, but only half so. I'm looking for some feedback on how simple it might be. It might be called the "CopyPasta Strainer" mod. The idea is to screen submitted posts for significant amounts of unattributed copy-pasted material. Simply grab a few sentences from the post, and do google searches on the entire sentences. The specifics can be varied, but once detected, the user would be prompted to edit their post and include a link within the post. (If the copy-pasted text is found in an included link, it passes).

[Antrax]

Not sure I understood what the question was. One thing I'd worry about is the possibility of DoSing the server by submitting multiple posts - waiting for the result of a Google query does take some time - especially if you can cause Google to throttle the server.
_________________
After years of disappointment with get rich quick schemes, I know I'm gonna get rich with this scheme. And quick!

[extropalopakettle]

Could the check be done on the client side? (I know it could then be worked around, but I'm assuming less sophisticated users and no real motive to do so)

[Zag]

The short answer is yes. The long answer is "OMG, What a headache!"

One certainly could write JavaScript code to make a REST query to Google and analyze the results before submitting. Doing this correctly cross-browser would be a challenge, though.

[Jack_Ian]

Perhaps adding a "Check for quotes" button would be easier. It could pass the text to a JavaScript routine that would run locally and display the results in another window.

[MNOWAX]

I'm not much of a programmer, but how would you do that type of stuff? Is javascript all that hard to play around with to do that type of button. I actually am thinking about doing that with my own site.
_________________
The Man The Myth The Legend
MNOWAX

[Chuck]

Have it check only sentences in which there are no spelling errors. That could save a great deal of time and bandwidth.

[extro...*]

There are a few boards I visit that have a few regular users who regularly copy-paste from far right sites ... the sort of users that I can tell they did so by the fact that there are grammatically correct sentences, and where when I google such a sentence I get hundreds of hits with the same right-wing blurb repeated on countless right-wing blogs. These users would never use a button, and can't be taught the simple notion that if you're copying someone else's words, you should give some explicit indication of that. It would just be nice to automatically tell such users "hey, this shit's been copied all over the net, don't post it here as if it's your own".

[Zag]

JavaScript is a pain because of the different versions. Probably, however, the code for a Google search returning JSON data and doing some text comparisons would be pretty safe for cross browser. (It is messing with the stuff on the page that has a lot of cross-browser issues.)

extro's original request, though, would be a lot harder than he thinks. If all you are looking to pick up is if the entire text of the post is exactly duplicated elsewhere on the 'net (perhaps more than so many times), that would be easy. You just quote the whole thing, require the exact quoted string, and ask Google how many hits it gets.

But something as simple as the person adding "Check this out" before the quote would break it. Duplicates that wetware would immediately see is a 98% copy is a hell of a challenge for a simple JavaScript program to pick out. Google would do pretty well, so that if you just sent the whole thing unquoted, probably the 98% exact copies would be clustered in the first bunch of hits, but Google has written a ton of code to make that happen. However, it doesn't help you to see that Google found thousands of hits; It'll find that many on any block of text. You'd have to analyze the first bunch of them to see if they are "close."

You could tweak the algorithm by skipping the first and last 10% of the post (rounding off to whole words), and then go back to requiring complete exact matches from Google to see how many it finds. That would fix the above problem and would probably accomplish the goal that extro has. I don't really see it as worth the trouble, though.

[Jack_Ian]

[Antrax]

Is it really that difficult, Zag? Just look for maximal matching substring and if its length > threshold, busted. To work around that they'd need to start changing words in the middle, which can at least be called "rephrasing"
_________________
After years of disappointment with get rich quick schemes, I know I'm gonna get rich with this scheme. And quick!

[Zag]

It depends whether or not you mind if the time between clicking the button and something happening to be sub-ten-minute or not.

There are a host of problems with that, Antrax. First, the Google results don't tell you the contents of the sites it found, so either you'd have to go to the many sites to get their content, or get the contents from Google's cached HTML. Second, even if you did get the content pretty quickly, it's not like JavaScript has any built-in functions for "maximal matching substring." You'd have to write it yourself, doing a whole lot of String operations: not JavaScript's strong suit.

[Antrax]

I think php has something to that effect.
_________________
After years of disappointment with get rich quick schemes, I know I'm gonna get rich with this scheme. And quick!

[Jack_Ian]

I still think trying to automate the process will produce far too many false positives. At minimum, you'd have to see 3 consecutive sentences being matched to have any confidence in the result.

[extro...*]

[Zag]

[Antrax]

Wait, what happened to moving everything to the cloud?
_________________
After years of disappointment with get rich quick schemes, I know I'm gonna get rich with this scheme. And quick!

[Zag]

I think that the term "Cloud Computing" is not well enough defined in the general public to have any meaning at all. Many people think of it simply as a common place to store and share documents. Others think of it more as the Google docs sort of thing, which is an interesting mix of fat client and server integration.

[Zahariel]

Javascript as a development language has had a few problems. The first is that, until 2 years ago or so, it couldn't be used for serious development because each browser implemented it slightly different, in a way that would break anything nontrivial. This didn't stop people from trying, of course, but it sure made lots of people angry with the entire concept, and it forced browsers to include the "turn off Javascript" option in order to protect their users from broken Javascript intended for some other browser (usually IE6), or to protect them from security problems in the browser's own Javascript implementation. Recently, certain historically noncompliant browsers have gotten their act together and made an actual attempt at implementing Javascript properly, and made tools like dojo and jQuery possible without insane workarounds.

The second problem was that Javascript's prototype-based object system is very foreign to a lot of people coming from a strong object-oriented background. More recent languages like Python and Lua have increased awareness of the benefits of systems of this nature, thus causing people to take another hard look at what Javascript can actually do, now that implementations are more reliable. Applications like Google Docs have been the result.

[Jack_Ian]

[MNOWAX]

fat client philosophy??
_________________
The Man The Myth The Legend
MNOWAX

[Zag]

A fat client architecture is where there is a lot of logic and code in the box on the user's desk (so to speak). In a thin client architecture, the logic and intelligence of the application is all applied on the server.

Originally, there were mainframes, and what a user interacted with was typewriter-like terminal that printed everything on a big spool of paper. It was the ultimate thin client architecture. Since the amount of actual data that traveled back and forth between the terminal and the computer was fairly small, you could actually be pretty far from the server, even over dial-up at 300 baud, and it would still be only a little painful to work with.

User input was, originally, a line at a time. i.e.

[Jack_Ian]

Adobe are trying to promote Flash in order to achieve this effect. The idea is to have all of the processing done on the client side in a consistent manner. This works well for games etc., but there is an annoying requirement for incessant upgrades from Adobe.
They operate assuming that users would just let it all happen automatically, but nobody with any sense would allow installations on their computer without some kind of warning, so these upgrades have become quite an annoyance.

[MNOWAX]

yeah but the casual user doesn't like the constant updating of Flash. That's the main problem Adobe is having.

The real question is how do you solve this problem?
_________________
The Man The Myth The Legend
MNOWAX

[Zag]

Easy. Plan a little more. Have better QA. And don't react immediately to every stupid little developer request.

One upgrade every 6 months would be fine. When some company is whining that they absolutely need that new feature that they asked for before they can ship their product, have the balls to hold tough on your feature list and your deadlines.

[MNOWAX]

yeah, but doyou think Flash will be the answer for for all coding in the near future?
_________________
The Man The Myth The Legend
MNOWAX

[Antrax]

[Zag]

That harkens back to my "Plan better. Have better QA." points.

None of those exploits were anything that they couldn't have anticipated. As soon as you give ability to write to the local hard drive, you know there's a potential for an exploit, so you approach it by tip-toeing into what you will allow rather than giving full access then trying to block the ones that are problems. It's not like they were goofy back-door combinations of features that would have been tricky to anticipate.

Speaking of which, the most amusing of these that I have ever heard of was in the Windows NT login page. They added a help button to it, which seems pretty harmless, right? It showed help for logging in (though if you need help in that, you probably need to get a computer from Apple, anyway). The button went to the standard help system, but pointing to a more limited help dictionary. However, with the standard help there is an option to re-index the dictionary, and there is an advanced option from there to choose a different dictionary. That step lets you walk the hard drive, seeing all the file names (which is enough of a security breach right there), but it turned out that the file which holds the encrypted passwords looked enough like a help dictionary that help would open it and show its contents. Since earlier encryptions on that file were not exactly RSA 128, they could be broken. Even with RSA 128, it turns out that you can break most people's passwords (at least, ten years ago) just by having a dictionary of what every English word, every name, and every date encrypts to and seeing if their encrypted password is in your dictionary.

[Jack_Ian]

Browsers are designed in a way to protect users from the web.
For example: Imagine you create a page allowing the user to click on a button that presents the user with a dialog to select a file for upload onto your site. There are many such sites, PhotoBucket etc. It is impossible for PhotoBucket, using just HTML and JavaScript to know the original location of the file on your hard-disk. That information is hidden by the browser in an opaque container. All you can do is tell the browser to start sending the contents of that thing pointed to by the container (wherever that may be).
This restriction is good, but it is still a restriction, and can prevent developers from doing things which the user might want.
To get around these restrictions, you can either write a browser plug-in yourself, which the user must install (not the safest option, since this plug-in could do nasty things to your computer) or make use of a less restrictive, trustworthy plug-in, such as Flash which users are more likely to install.
The trouble is, since Flash is installed behind the security curtain, it can now circumvent the security inherent in the standard browser, and since it is still in a state of flux as it is under constant development, it is naturally full of security holes.
Adobe is in a real battle to attain dominance in this realm and cannot afford to be labelled as a security risk, which would spell the end, especially since the iPhone flat out refused to support it. All it takes is for some virus to spread to android phones via Flash and the Flash product will be scrapped, or at least re-branded and re-engineered.

[Zag]

My point exactly. Adobe should have built in the same security that is built into the HTML file upload tool. Did you know that it is the one element type which can not be set via JavaScript? That's part of my tip-toeing into the functionality rather than giving full access then trying to block what you don't want.

I would have started by having the user, upon install of Flash, to specify a "Flash Transfer" directory. The upload tool could only load from there, and the directory selection can't be changed from inside a Flash program. Part of the install would be to add the Flash Transfer directory to the "Send To" list in Windows.

Meanwhile, I'd create a file, the size of which the user can limit, and I'd manage the saving and loading of data for all Flash programs, and it only goes into that file. (They do this, actually.) I'd give no access to the core API's in the system.

This would create some grumbling from both developers and users, and I might cautiously and gradually allow some other special-case access. However, so far I guarantee no exploits are possible without the user stupidly agreeing to something. The most likely exploit is that hackers would make executables that claim to be a Flash update, but that is simply impossible to prevent other than training users not to believe every stupid thing they read.

[Jack_Ian]

I had some similar problems trying to manipulate drag&Drop flavours. I just couldn't get the required flavours added in a way that they persisted outside of the browser. Became a real pain. I had the option of using Flex, but the user would never have gone for it.
In then end I had to write plug-ins for InDesign and Quark in order for them to handle the standard browser flavours. Of course this means that I now have to do something similar for any other application we intend to support. Meh!

[MNOWAX]

What do you Do Jack? What is your background?
_________________
The Man The Myth The Legend
MNOWAX

[extropalopakettle]

Example:

http://www.greylabyrinth.com/discussion/viewtopic.php?t=6810&postdays=0&postorder=asc&start=449

[Jack_Ian]

[Lucresia]

I am not a programmer (I work with VoIP and telecom stuff), but I did find this whole thread very interesting to read. Especially Zag's post on IBM and fat client architecture. I also really like your idea extro and hope you are able to implement it!

(edited cause I wrote expo instead of extro..heh)
_________________
"And All anybody knows, is you're not like them" -Elliott Smith

[Zag]

Thanks Lucresia.

From 1992 to 1995 I worked at Ziff-Davis Interchange, and we were making a product that would compete with Prodigy and Compuserve. We actually had our beta and were up to, I think, 30,000 users. (I had the username Z.) It was the ultimate fat client approach (which I designed), but it included a self-update feature to avoid the common complaint of fat clients. (It's rather common today, but was not very common in the early 90's. We didn't invent the idea, though.)

The product was really good, highly graphical and extremely fast (over a 9600 baud modem!) thanks to a caching algorithm designed by yours truly. (I could go on about why it was so effective, but no one but me would be interested.) When we released our first beta in early 1994, we were easily the hit of the computer conference called "Demo 94" which was taking over as the computer software version of the Consumer Electronics Show (which was THE show for many years). We introduced a number of new concepts, including the one we're using here -- a discussion board where you see a whole thread (that is, different people's posts) at once.

Ziff-Davis, at the time, was the number one publisher of Computer magazines, and already had the products ZiffNet for Compuserve and ZiffNet for Prodigy. However, those platforms (Compuserve and Prodigy) were designed by computer nerds without really a thought of how to make money with it. When they tried to make money with them, it was a heavy-handed approach that annoyed the customers.

Ziff Interchange was designed from the point of view of a magazine publisher. Magazines are terrific at making money while making customers happy. I have actually had the 3-day seminar from Bill Ziff himself on "What is a Magazine?" (Short answer: a vehicle for delivering readers to advertisers. Not anything else, except for some very specialized and very expensive magazines which don't have advertisements.) In the course of developing Interchange, I became a reasonably friendly acquaintance of Bill's son, Robert Ziff. This was cool because I had never before had a friend whose net worth had three commas in it. (The family together exceeded Bill Gates for a while, but no one of them ever did.)

Unfortunately, the juggernaut of the Internet crushed our product. In spite of how poorly designed and inefficient it is, how incredibly painful it was to use on a 9600 baud modem, etc. etc. we were steamrolled. The market never does go with the best technology.

[Antrax]

[extro...*]

[extro...*]

I was asked in a job interview to write java code to implement a caching algorithm with a fixed size cache, where removal of elements for new ones was by least recently used, and where insertion and lookup were constant time operations. I nailed it. Didn't get the job though ... I didn't react well to the weird psychological questions. Kinda just gave them an honest "are you for real?" look. I enjoyed the tech questions though.

[Scurra]

During the 90s I hung out a lot on a UK bulletin board service called CIX (which is still going, amazingly!)
The AMEOL software that was developed there 20-odd years ago is still fantastic, and makes all bar the most dedicated forum software look amateurish. I find it extraordinary that people actually put up with the useless systems that e.g. WordPress provide, even today.
_________________
_{still Quiz Olympiad champion. Must get a life.
New definitions: COFFEE - someone who is coughed upon}