When we think about music people often understand things at an intuitive level but become uneasy about the idea of using rules or recipes to generate it. "Formulaic" is a dirty word when used to describe pop songs for example, but it doesn't have to be that way. I'm going to look at one particular algorithm, Euclid's algorithm, to create musical rhythms.

A method for spacing out rhythms

Despite all the complex words I'm about to use, if all boils down to this: if I have a number of beats that I want to play in a given space (say 2 bars), how do I space those out as evenly as possible?

That's obviously easy with regular rhythms that divide cleanly. If I want to play 8 beats across two bars of 8th notes it would look like this:

x . x . x . x . | x . x . x . x .  

That's 8 x's spaced across the 16 possible beats. Easy peasy.

But what about numbers that don't divide cleanly, for example, 3 into 8?

Let's start with the solution first:

x . . x . . x .  

You can see we have to use a combination of long and short notes to fill the bar evenly. The above pattern uses groups of 3's and 2's to fill the space. You could easily notate it as:

332  

if you count the distances from one beat to the next. Let's look at a few more examples of these odd rhythms:

# 5 into 8
x . x . x x . x  
22121

# 13 into 16
x . x x x x . x | x x x . x x x x  
2111211121111  

One thing you might notice is that these aren't the only way to divide up the bar. To take our 3 into 8 rhythm there are actually three possible options:

x . . x . . x . #== 332  
x . x . . x . . #== 233  
x . . x . x . . #== 323  

The last of those options appears so often in so many kinds of music that it's often called "the mother rhythm". Everything from Cuban clave to Elvis' "Hound dog". That said, if you look closely they are all variations on the same grouping: 2 lots of 3 and one lot of 2. That means we only need to generate one version and then we can rotate it to produce the others. So how are we going to generate it then?

Back to the future: Euclid's algorithm from 300 B.C.

It turns out that Euclid thought about this too - how do you space things out as evenly as possible? Thousands of years later a scientist named Bjorklund was facing the same problem when he was setting up a Neutron Accelerator. He needed to fire neutrons evenly in a given space of time (a bit like our rhythm problem) and, being a scientist, he came up with a clever way of doing that.

    def distribute(accents, total_beats)
      res = []

      total_beats.times do |i|
        # makes a boolean based on the index
        # true is an accent, false is a rest
        res << ((i * accents % total_beats) < accents)
      end

      res
    end

That's the algorithm represented in Ruby. You don't have to understand it as it's probably not the simplest way of explaining it, but it's there to show how concise a problem like this can be when expressed in code.

Years later a percussion-playing, scientifically-minded music professor was looking at the same problem applied to musical rhythms and came across Bjorklund's research. He took the idea and explored it in a paper "The Euclidean Algorithm Generates Traditional Musical Rhythms" (Toussaint 2005) from which I've taken these ideas.

In Sonic Pi...

I've added a function to Sonic Pi called distribute which allows you to generate arrays that fit these evenly spaced patterns. Here's some sample code:

def distribute(accents, total_beats, beat_rotations=0)  
  res = []

  total_beats.times do |i|
    # makes a boolean based on the index
    # true is an accent, false is a rest
    res << ((i * accents % total_beats) < accents)
  end

  res.ring
end

class SonicPi::Core::RingArray  
  def as_x_notation
    self.to_a.map {|x| x ? 'x' : '.'}.join(' ')
  end

  def as_beat_groups
    self.to_a.slice_before {|x| x }.to_a.map(&:count).join
  end
end

puts distribute(3, 8, 0).inspect  
puts distribute(3, 8, 0).as_beat_groups  
puts distribute(3, 8, 0).as_x_notation  

If you paste in the above code to SonicPi and run it, you should see this in the output window:

 ├─ [true, false, false, true, false, false, true, false]
 ├─ 332
 └─ x . . x . . x .

From here it's not too hard to map over those arrays to turn them into musical patterns. Here's an example you can play with:

comment do
  # this function will be available in SonicPi v2.4 and upwards
  # if you're using a version with this included you can 
  # delete this distribute function, otherwise uncomment it
  # above
  def distribute(accents, total_beats, beat_rotations=0)
    res = []

    total_beats.times do |i|
      # makes a boolean based on the index
      # true is an accent, false is a rest
      res << ((i * accents % total_beats) < accents)
    end

    res.ring
  end
end

# Monkey patching is never a good idea
# just say no kids...
class SonicPi::Core::RingArray
  def as_x_notation
    self.to_a.map {|x| x ? 'x' : '.'}.join(' ')
  end

  def as_beat_groups
    self.to_a.slice_before {|x| x }.to_a.map(&:count).join
  end
end

use_bpm(120)

def play_sample_for_sequence(pattern, sample_name, sleep_time = 0.25)
  pattern.each.with_index do |beat, i|
    sample sample_name if !!beat
    sleep sleep_time
  end
end

uncomment do
  live_loop(:hh) do
    with_fx :level, amp: 1 do
      cue :heartbeat
      play_sample_for_sequence(distribute(11, 16), :drum_cymbal_closed)
    end
  end

  live_loop(:bd) do
    sync :heartbeat
    play_sample_for_sequence(distribute(5, 16), :drum_bass_hard)
  end

  live_loop(:sn) do
    with_fx :level, amp: 1 do
      cue :heartbeat
      play_sample_for_sequence(distribute(2, 16).to_a.rotate(4), :drum_snare_hard)
    end
  end

  live_loop(:bass) do
    with_fx :level, amp: 1 do
      use_synth :tb303
      cue :heartbeat
      distribute(3, 8).each do |beat|
        play scale(:a2, :minor_pentatonic).choose, release: 0.3 if beat
        sleep 0.5
      end
    end
  end
end  

Here's a demo of me live coding with the above:

Why does this work?

Musically I think it's fair to say that last example sounds pretty cool. But why? I think that the appeal of these rhythms ("Euclidean Rhythms" as Toussaint calls them) boils down to this: because the spacing is always even, when we layer these kinds of beats on top of each other we get strong "cross rhythms". In the example above I've deliberately put the snare on beats 2 and 4 to provide a strong backbeat. All the other rhythms bounce off the strength of the snare rhythm.

I think another possible reason is that our brains enjoy patterns and order even if we can't quite tell what the pattern is. Imagine the difference between two knitted jumpers, one with a completely random colour for every stitch, the other with a strong geometric pattern. Psychologically we're more likely to prefer the geometric pattern (unless the random knit happens to be really cool).

Having these functions in our toolkit means that we can reach musical sounding results even faster, without having to rely on things being totally random but still retaining an element of surprise.

What do you think? Does it sound cool to you? Will algorithmic music ever be widely accepted? Hit me up on twitter @xavriley if you have any comments or questions.

Further rhythming

This topic has already been covered in several other places, but not as far as I know in Sonic Pi yet. There's a very cool HTML5/Javascript version of this kind of drum machine available at http://www.groovemechanics.com/euclid/ which has been on Hacker news. Also there's a patch for Max called Polyrhythmus which is along the same lines. I wanted to cover the musical aspects behind the algorithm in a bit more depth for this post. There's also a good Wikipedia page on Euclidean rhythm with links to other resources.

Exhibit A: Steve Klabnik on Open Source

Open source is a mega-capitalist conspiracy to reduce programmer wages. ... it's yet another method to drive down costs. ...

The other day I was hanging out with two people and there was somebody there using a jackhammer outside where we were. I said, "Nobody asked the jackhammer dude to work on an open jackhammer project on Saturdays, that would be totally absurd." But that's what we do. Programmers as a class systematically devalue their own labor, doing it for free without being asked to.

It's not universally bad - it's really cool that you can get all this software for free. But, value is being generated from your labor and it's not being captured by you, and that's unsustainable. I'm only able to do open source work now because I'm getting a real salary from Mozilla. They're a one-of-a-kind organization that can afford to pay people for open source stuff ... but the whole world can't work for Mozilla.

Excerpt taken from "Steve Klabnik: Programming for Programmers" in Mask Magazine - the cyborgoisie issue NB subscription required

He goes on to say how an open source portfolio is now a key job selection criteria which is setting the balance in favour of those who have the free time to work on those projects.

wat?

Here we have one of the most prolific Open Source contributors of all time calling out Open Source as a (mostly) bad idea. Particularly when viewed from a marxist standpoint. This begs a questions about when and where "Open Source" steps over the line into being bad, which I'll try to answer.

Exhibit B: Heather Gold - "Designing for conversation"

Nothing brings people together like a shared problem. This is what holds people together. Part of the reason we solve problems isn't just to build a bridge to get across the river - it's because the feeling of doing it together is awesome. That's probably one of the best feelings we have in life.

Heather Gold - "Designing for Conversation"

If we can agree on this premise, I think it's the best explanation of why we as programmers choose to give up our labour for free. Collaborating on a shared problem is awesome. It feels great. Github is built on the fact that having a pull request accepted, or code you've written being used by others, is a universally good feeling. But this feeling is open to abuse when other forces get involved.

Evil #1 - the "awesome feeling" gets co-opted by evil actors

Here I'm mainly talking about corporate hack days. A while ago, I saw a tweet about Unilever running a weekend hackathon. This upset some people and ISTR it making it onto Hacker News.

“I know what I’ll do with my weekend, I’ll sit in a room and code so Unilever can add a few points to their share value” Said no one ever

The problem here is that Unilever have a market cap of around $74 Billion dollars. They could afford to pay every single attendee of that hackathon a handsome wage for their time and ideas and yet they don't.

As programmers, we need to become aware of these issues like Klabnik is telling us to. Hackathons are not evil by nature - at least with doing a hackathon for charity or the government where you can cite some feeling of civic or moral duty. Even at a stretch, a corporate hackathon might be worthwhile if it presents some valuable opportunity for learning. I can fully see that if I was interested in learning how email tech works at scale, attending a hackathon hosted by SendGrid would be a good use of my time. The same can't be said for Unilever in my opinion. If you're attending a hackathon or contributing to an open source project ask yourself - "do these people value my input with the proper respect?"

Evil #2 - we programmers lose sight of the "awesome feeling"

A topical example. Nick Sutterer is the developer of the popular Active Form plugin for Rails. He recently tweeted this:

https://twitter.com/apotonick/status/517840615866769408

What appears to have happened is that some other developer has decided Active Form is better than the current defaults in Rails and copied the project wholesale without attribution. The incentive to be a "core rails" committer is a strong one and I can see that, for those without a moral compass, stealing code from another established project is a tempting option. The result? Another dev has "sold out" @apotonick in order to improve their rep in the Open Source world.

What's interesting to me is that some of the comments appear to be questioning this action on the technicalities of the licence involved. That might be appropriate in a commercial transaction, but in an act of social collaboration? I'm not sure. Everyone is giving their time for free and that needs to be respected - the other dev has denied @apotonick the feeling of pride and satisfaction that would have come from seeing his project merged into Rails. We should all learn from this example - the "Open Source spirit" is built on a bedrock of attributing people properly for their hard work. If we don't call this kind of behaviour out then the whole thing will crumble.

Evil #3 - Open Source != sustainable

As a rule, working for free isn't sustainable. That said, in an ideal world we all have a bit of leisure time to spend as we please. If I feel like contributing to an awesome Open Source music programming project and I have the time, that's my perogative. I think this third evil arises when the motivations for contributing get mixed up. When we're not clear about the boundary between hobby projects and work projects things start to come unstuck. If I start out writing a new super fast web framework as a hobby project (perhaps to learn more or because I thought I could do better than what's out there) that's fine. But let's say that project gets picked up and used by thousands of businesses around the world. This project has made the leap from bedroom hacking to critical infrastructure - success? It is a success of sorts. The good feeling of having your contribution used in the world is undeniable. However, Open Source doesn't have a notion of responsibility here - it's quite likely that people will hold you personally responsible for your projects bugs and failures.

Here's a real world example of TJ Holowaychuk who was so prolific that this Quora post questions whether he really exists. He wrote a massive amount of widely used node js infrastructure and then left it to the community to maintain. To reiterate, that's his right to spend his time as he pleases and there's nothing wrong with that. Perhaps he needed to take a paying job? Whatever the reason, he's one of a string of dropouts from the Open Source community which suggests that burnout is a real hazard both to the individual and to the businesses that rely on their work.

What's the solution here? Speaking with some tech people at Wikipedia they seemed to be intrinsically aware of this. Their tools for wikimedia users and developers are built on the premise that if it becomes valuable infrastructure then the Wikimedia foundation will shoulder the responsibility if you choose to abandon the project. For this reason they require that you host on their infrastructure (for free) and publish the source of your work. This helps to take some of the heat off.

Another solution might be GitTip and other ways of recognizing contributions financially. Whilst these are good, I suspect that they lead to a real world undervaluation of a person's contribution.

The last point is that for Open Source to be sustainable, it should probably be sponsored or supported by an organization. The idea of Wordpress contributing back to ElasticSearch through open source is an entirely natural one. Joint ventures and partnerships are common in the business world. Steve Klabnik is supported by Mozilla and appears to be comfortable with this arrangment. I think that as developers, we should develop a sense of which projects are appropriate for hacking on personally and which are better suited to some kind of organisational sponsorship. Secondly we need to be honest about this, both with the community and ourselves.

Why PDFTextStream?

Quite simply, it's the best PDF extraction library I've come across in terms of features and performance. It handles layouts and formatting very well and the xml output gives some useful tags for data extraction.

Getting the library

Head over to http://snowtide.com/downloads and download the latest Java version (2.7.0 at the time of writing) and unzip into a folder called jruby-demo.

Some JRuby/Java interop

Create a file at jruby-demo/pdf-extractor.rb with the following contents:

require 'java'  
require 'json'  
require 'PDFTextStream.Java-2.7.0/lib/PDFTextStream.jar'  
$CLASSPATH << 'PDFTextStream.Java-2.7.0/src'

java_import com.snowtide.pdf.PDFTextStream  
java_import com.snowtide.pdf.OutputTarget # To output plain text  
java_import "pdfts.examples.XMLOutputTarget" # To output XML  
java_import java.lang.StringBuilder

pdf_file_path = File.join(Dir.pwd, ARGV[0])

  sb = StringBuilder.new # Requires Java StringBuilder for some reason
  pdfts = Java::ComSnowtidePdf::PDFTextStream.new(pdf_file_path)

  case ARGV[1]
    when "xml"
      # XMLOutputTarget keeps the formatting tags of 
      # the input PDF - useful if the source uses bold or italics etc.
      ot = XMLOutputTarget.new

      pdfts.pipe(ot)
      pdfts.close
      puts ot.getXMLAsString
    when "standard"
      # Normal OutputTarget reformats the text to handle
      # column layouts
      ot = Java::ComSnowtidePdf::OutputTarget.new(sb)

      pdfts.pipe(ot)
      pdfts.close
      puts sb.to_s
    when "visual"
      # VisualOutputTarget is better at preserving layout in
      # the conversion to text e.g. tables
      ot = Java::ComSnowtidePdf::VisualOutputTarget.new(sb)

      pdfts.pipe(ot)
      pdfts.close
      puts sb.to_s
    else
      # VisualOutputTarget is better at preserving layout in
      # the conversion to text e.g. tables
      ot = Java::ComSnowtidePdf::VisualOutputTarget.new(sb)

      pdfts.pipe(ot)
      pdfts.close
      puts sb.to_s
  end

Extracting some text

Move a pdf into the folder, install jruby and then run:

cd jruby-demo  
jruby pdf-extractor.rb name-of-your-pdf.pdf standard  

and after a few seconds of jvm warm up time you should start to see text on STDOUT.

Different extraction modes

standard - This handles column layouts (common in pdfs) and reflows them to make sure the text reads in the correct order.

visual - This preserves the text spacing on the page which is useful for tabular data.

xml - If the source data has bold or italic text, this processor outputs xml markup which can be useful for further processing with Nokogiri or other similar libraries.

Source pdf available at http://www.europarl.europa.eu/document/activities/cont/201403/20140307ATT80674/20140307ATT80674EN.pdf

Markdown available here: https://dl.dropboxusercontent.com/u/11192828/edward_snowden_testimony.md

Introductory Statement

I would like to thank the European Parliament for the invitation to provide testimony for your inquiry into the Electronic Mass Surveillance of EU Citizens. The suspicionless surveillance programs of the NSA, GCHQ, and so many others that we learned about over the last year endanger a number of basic rights which, in aggregate, constitute the foundation of liberal societies.

The first principle any inquiry must take into account is that despite extraordinary political pressure to do so, no western government has been able to present evidence showing that such programs are necessary. In the United States, the heads of our spying services once claimed that 54 terrorist attacks had been stopped by mass surveillance, but two independent White House reviews with access to the classified evidence on which this claim was founded concluded it was untrue, as did a Federal Court.

Looking at the US government's reports here is valuable. The most recent of these investigations, performed by the White House's Privacy and Civil Liberties Oversight Board, determined that the mass surveillance program investigated was not only ineffective -- they found it had never stopped even a single imminent terrorist attack -- but that it had no basis in law. In less diplomatic language, they discovered the United States was operating an unlawful mass surveillance program, and the greatest success the program had ever produced was discovering a taxi driver in the United States transferring $8,500 dollars to Somalia in 2007.

After noting that even this unimpressive success -- uncovering evidence of a single unlawful bank transfer -- would have been achieved without bulk collection, the Board recommended that the unlawful mass surveillance program be ended. Unfortunately, we know from press reports that this program is still operating today.

I believe that suspicionless surveillance not only fails to make us safe, but it actually makes us less safe. By squandering precious, limited resources on "collecting it all," we end up with more analysts trying to make sense of harmless political dissent and fewer investigators running down real leads. I believe investing in mass surveillance at the expense of traditional, proven methods can cost lives, and history has shown my concerns are justified.

Despite the extraordinary intrusions of the NSA and EU national governments into private communications world-wide, Umar Farouk Abdulmutallab, the "Underwear Bomber," was allowed to board an airplane traveling from Europe to the United States in 2009. The 290 persons on board were not saved by mass surveillance, but by his own incompetence, when he failed to detonate the device. While even Mutallab's own father warned the US government he was dangerous in November 2009, our resources were tied up monitoring online games and tapping German ministers. That extraordinary tip-off didn't get Mutallab a dedicated US investigator. All we gave him was a US visa.

Nor did the US government's comprehensive monitoring of Americans at home stop the Boston Bombers. Despite the Russians specifically warning us about Tamerlan Tsarnaev, the FBI couldn't do more than a cursory investigation -- although they did plenty of worthless computer-based searching - and failed to discover the plot. 264 people were injured, and 3 died. The resources that could have paid for a real investigation had been spent on monitoring the call records of everyone in America.

This should not have happened. I worked for the United States' Central Intelligence Agency. The National Security Agency. The Defense Intelligence Agency. I love my country, and I believe that spying serves a vital purpose and must continue. And I have risked my life, my family, and my freedom to tell you the truth.

The NSA granted me the authority to monitor communications world-wide using its mass surveillance systems, including within the United States. I have personally targeted individuals using these systems under both the President of the United States' Executive Order 12333 and the US Congress' FAA 702. I know the good and the bad of these systems, and what they can and cannot do, and I am telling you that without getting out of my chair, I could have read the private communications of any member of this committee, as well as any ordinary citizen. I swear under penalty of perjury that this is true.

These are not the capabilities in which free societies invest. Mass surveillance violates our rights, risks our safety, and threatens our way of life.

If even the US government, after determining mass surveillance is unlawful and unnecessary, continues to operate to engage in mass surveillance, we have a problem. I consider the United States Government to be generally responsible, and I hope you will agree with me. Accordingly, this begs the question many legislative bodies implicated in mass surveillance have sought to avoid: if even the US is willing to knowingly violate the rights of billions of innocents -- and I say billions without exaggeration -- for nothing more substantial than a "potential" intelligence advantage that has never materialized, what are other governments going to do?

Whether we like it or not, the international norms of tomorrow are being constructed today, right now, by the work of bodies like this committee. If liberal states decide that the convenience of spies is more valuable than the rights of their citizens, the inevitable result will be states that are both less liberal and less safe.

Thank you.

I will now respond to the submitted questions. Please bear in mind that I will not be disclosing new information about surveillance programs: I will be limiting my testimony to information regarding what responsible media organizations have entered into the public domain. For the record, I also repeat my willingness to provide testimony to the United States Congress, should they decide to consider the issue of unconstitutional mass surveillance.

Rapporteur Claude Moraes MEP, S&D Group

Given the focus of this Inquiry is on the impact of mass surveillance on EU citizens, could you elaborate on the extent of cooperation that exists between the NSA and EU Member States in terms of the transfer and collection of bulk data of EU citizens?

A number of memos from the NSA's Foreign Affairs Directorate have been published in the press.

One of the foremost activities of the NSA's FAD, or Foreign Affairs Division, is to pressure or incentivize EU member states to change their laws to enable mass surveillance. Lawyers from the NSA, as well as the UK's GCHQ, work very hard to search for loopholes in laws and constitutional protections that they can use to justify indiscriminate, dragnet surveillance operations that were at best unwittingly authorized by lawmakers. These efforts to interpret new powers out of vague laws is an intentional strategy to avoid public opposition and lawmakers’ insistence that legal limits be respected, effects the GCHQ internally described in its own documents as "damaging public debate."

In recent public memory, we have seen these FAD "legal guidance" operations occur in both Sweden and the Netherlands, and also faraway New Zealand. Germany was pressured to modify its G-10 law to appease the NSA, and it eroded the rights of German citizens under their constitution. Each of these countries received instruction from the NSA, sometimes under the guise of the US Department of Defense and other bodies, on how to degrade the legal protections of their countries' communications. The ultimate result of the NSA's guidance is that the right of ordinary citizens to be free from unwarranted interference is degraded, and systems of intrusive mass surveillance are being constructed in secret within otherwise liberal states, often without the full awareness of the public.

Once the NSA has successfully subverted or helped repeal legal restrictions against unconstitutional mass surveillance in partner states, it encourages partners to perform “access operations.” Access operations are efforts to gain access to the bulk communications of all major telecommunications providers in their jurisdictions, normally beginning with those that handle the greatest volume of communications. Sometimes the NSA provides consultation, technology, or even the physical hardware itself for partners to "ingest" these massive amounts of data in a manner that allows processing, and it does not take long to access everything. Even in a country the size of the United States, gaining access to the circuits of as few as three companies can provide access to the majority of citizens' communications. In the UK, Verizon, British Telecommunications, Vodafone, Global Crossing, Level 3, Viatel, and Interoute all cooperate with the GCHQ, to include cooperation beyond what is legally required. http://www.theguardian.com/business/2013/aug/02/telecoms-bt-vodafone-cables-gchq

By the time this general process has occurred, it is very difficult for the citizens of a country to protect the privacy of their communications, and it is very easy for the intelligence services of that country to make those communications available to the NSA -- even without having explicitly shared them. The nature of the NSA's "NOFORN," or NO FOREIGN NATIONALS classification, when combined with the fact that the memorandum agreements between NSA and its foreign partners have a standard disclaimer stating they provide no enforceable rights, provides both the NSA with a means of monitoring its partner's citizens without informing the partner, and the partner with a means of plausible deniability.

The result is a European bazaar, where an EU member state like Denmark may give the NSA access to a tapping center on the (unenforceable) condition that NSA doesn't search it for Danes, and Germany may give the NSA access to another on the condition that it doesn't search for Germans. Yet the two tapping sites may be two points on the same cable, so the NSA simply captures the communications of the German citizens as they transit Denmark, and the Danish citizens as they transit Germany, all the while considering it entirely in accordance with their agreements. Ultimately, each EU national government's spy services are independently hawking domestic accesses to the NSA, GCHQ, FRA, and the like without having any awareness of how their individual contribution is enabling the greater patchwork of mass surveillance against ordinary citizens as a whole.

The Parliament should ask the NSA and GCHQ to deny that they monitor the communications of EU citizens, and in the absence of an informative response, I would suggest that the current state of affairs is the inevitable result of subordinating the rights of the voting public to the prerogatives of State Security Bureaus. The surest way for any nation to become subject to unnecessary surveillance is to allow its spies to dictate its policy.

The right to be free unwarranted intrusion into our private effects -- our lives and possessions, our thoughts and communications -- is a human right. It is not granted by national governments and it cannot be revoked by them out of convenience. Just as we do not allow police officers to enter every home to fish around for evidence of undiscovered crimes, we must not allow spies to rummage through our every communication for indications of disfavored activities.

Could you comment on the activities of EU Member States intelligence agencies in these operations and how advanced their capabilities have become in comparison with the NSA?

The best testimony I can provide on this matter without pre-empting the work of journalists is to point to the indications that the NSA not only enables and guides, but shares some mass surveillance systems and technologies with the agencies of EU member states. As it pertains to the issue of mass surveillance, the difference between, for example, the NSA and FRA is not one of technology, but rather funding and manpower. Technology is agnostic of nationality, and the flag on the pole outside of the building makes systems of mass surveillance no more or less effective.

In terms of the mass surveillance programmes already revealed through the press, what proportion of the mass surveillance activities do these programmes account for? Are there many other programmes, undisclosed as of yet, that would impact on EU citizens rights?

There are many other undisclosed programs that would impact EU citizens' rights, but I will leave the public interest determinations as to which of these may be safely disclosed to responsible journalists in coordination with government stakeholders.

Shadow Rapporteur Sophie Int'Veld MEP, ALDE Group

Are there adequate procedures in the NSA for staff to signal wrongdoing?

Unfortunately not. The culture within the US Intelligence Community is such that reporting serious concerns about the legality or propriety of programs is much more likely to result in your being flagged as a troublemaker than to result in substantive reform. We should remember that many of these programs were well known to be problematic to the legal offices of agencies such as the GCHQ and other oversight officials. According to their own documents, the priority of the overseers is not to assure strict compliance with the law and accountability for violations of law, but rather to avoid, and I quote, "damaging public debate," to conceal the fact that for-profit companies have gone "well beyond" what is legally required of them, and to avoid legal review of questionable programs by open courts. (http://www.theguardian.com/uk- news/2013/oct/25/leaked-memos-gchq-mass-surveillance-secret-snowden)

In my personal experience, repeatedly raising concerns about legal and policy matters with my co-workers and superiors resulted in two kinds of responses.

The first were well-meaning but hushed warnings not to "rock the boat," for fear of the sort of retaliation that befell former NSA whistleblowers like Wiebe, Binney, and Drake. All three men reported their concerns through the official, approved process, and all three men were subject to armed raids by the FBI and threats of criminal sanction. Everyone in the Intelligence Community is aware of what happens to people who report concerns about unlawful but authorized operations.

The second were similarly well-meaning but more pointed suggestions, typically from senior officials, that we should let the issue be someone else's problem. Even among the most senior individuals to whom I reported my concerns, no one at NSA could ever recall an instance where an official complaint had resulted in an unlawful program being ended, but there was a unanimous desire to avoid being associated with such a complaint in any form.

Do you feel you had exhausted all avenues before taking the decision to go public?

Yes. I had reported these clearly problematic programs to more than ten distinct officials, none of whom took any action to address them. As an employee of a private company rather than a direct employee of the US government, I was not protected by US whistleblower laws, and I would not have been protected from retaliation and legal sanction for revealing classified information about lawbreaking in accordance with the recommended process.

It is important to remember that this is legal dilemma did not occur by mistake. US whistleblower reform laws were passed as recently as 2012, with the US Whistleblower Protection Enhancement Act, but they specifically chose to exclude Intelligence Agencies from being covered by the statute. President Obama also reformed a key executive Whistleblower regulation with his 2012 Presidential Policy Directive 19, but it exempted Intelligence Community contractors such as myself. The result was that individuals like me were left with no proper channels.

Do you think procedures for whistleblowing have been improved now?

No. There has not yet been any substantive whistleblower reform in the US, and unfortunately my government has taken a number of disproportionate and persecutory actions against me. US government officials have declared me guilty of crimes in advance of any trial, they've called for me to be executed or assassinated in private and openly in the press, they revoked my passport and left me stranded in a foreign transit zone for six weeks, and even used NATO to ground the presidential plane of Evo Morales - the leader of Bolivia - on hearing that I might attempt to seek and enjoy asylum in Latin America.

What is your relationship with the Russian and Chinese authorities, and what are the terms on which you were allowed to stay originally in Hong Kong and now in Russia?

I have no relationship with either government.

Shadow Rapporteur Jan Philipp Albrecht MEP, Greens Group

Could we help you in any way, and do you seek asylum in the EU?

If you want to help me, help me by helping everyone: declare that the indiscriminate, bulk collection of private data by governments is a violation of our rights and must end. What happens to me as a person is less important than what happens to our common rights.

As for asylum, I do seek EU asylum, but I have yet to receive a positive response to the requests I sent to various EU member states. Parliamentarians in the national governments have told me that the US, and I quote, "will not allow" EU partners to offer political asylum to me, which is why the previous resolution on asylum ran into such mysterious opposition. I would welcome any offer of safe passage or permanent asylum, but I recognize that would require an act of extraordinary political courage.

Can you confirm cyber-attacks by the NSA or other intelligence agencies on EU institutions, telecommunications providers such as Belgacom and SWIFT, or any other EU-based companies?

Yes. I don't want to outpace the efforts of journalists, here, but I can confirm that all documents reported thus far are authentic and unmodified, meaning the alleged operations against Belgacom, SWIFT, the EU as an institution, the United Nations, UNICEF, and others based on documents I provided have actually occurred. And I expect similar operations will be revealed in the future that affect many more ordinary citizens.

Shadow Rapporteur Cornelia Ernst MEP, GUE Group

In your view, how far can the surveillance measures you revealed be justified by national security and from your experience is the information being used for economic espionage? What could be done to resolve this?

Surveillance against specific targets, for unquestionable reasons of national security while respecting human rights, is above reproach. Unfortunately, we've seen a growth in untargeted, extremely questionable surveillance for reasons entirely unrelated to national security. Most recently, the Prime Minister of Australia, caught red-handed engaging in the most blatant kind of economic espionage, sought to argue that the price of Indonesian shrimp and clove cigarettes was a "security matter." These are indications of a growing disinterest among governments for ensuring intelligence activities are justified, proportionate, and above all accountable. We should be concerned about the precedent our actions set.

The UK's GCHQ is the prime example of this, due to what they refer to as a "light oversight regime," which is a bureaucratic way of saying their spying activities are less restricted than is proper (http://www.theguardian.com/uk/2013/jun/21/legal-loopholes-gchq-spy-world). Since that light oversight regime was revealed, we have learned that the GCHQ is intercepting and storing unprecedented quantities of ordinary citizens' communications on a constant basis, both within the EU and without http://www.theguardian.com/uk/2013/jun/21/gchq-cables-secret- world-communications-nsa). There is no argument that could convince an open court that such activities were necessary and proportionate, and it is for this reason that such activities are shielded from the review of open courts.

In the United States, we use a secret, rubber-stamp Foreign Intelligence Surveillance Court that only hears arguments from the government. Out of approximately 34,000 government requests over 33 years, the secret court rejected only 11. It should raise serious concerns for this committee, and for society, that the GCHQ's lawyers consider themselves fortunate to avoid the kind of burdensome oversight regime that rejects 11 out of 34,000 requests. If that's what heavy oversight looks like, what, pray tell, does the GCHQ's "light oversight" look like?

Let's explore it. We learned only days ago that the GCHQ compromised a popular Yahoo service to collect images from web cameras inside citizens' homes, and around 10% of these images they take from within people's homes involve nudity or intimate activities (http://www.theguardian.com/world/2014/feb/27/gchq-nsa-webcam-images-internet-yahoo). In the same report, journalists revealed that this sort of webcam data was searchable via the NSA's XKEYSCORE system, which means the GCHQ's "light oversight regime" was used not only to capture bulk data that is clearly of limited intelligence value and most probably violates EU laws, but to then trade that data with foreign services without the knowledge or consent of any country's voting public.

We also learned last year that some of the partners with which the GCHQ was sharing this information, in this example the NSA, had made efforts to use evidence of religious conservatives' association with sexually explicit material of the sort GCHQ was collecting as a grounds for destroying their reputations and discrediting them (http://www.huffingtonpost.com/2013/11/26/nsa-porn-muslimsn4346128.html). The "Release to Five Eyes" classification of this particular report, dated 2012, reveals that the UK government was aware of the NSA's intent to use sexually explicit material in this manner, indicating a deepening and increasingly aggressive partnership. None of these religious conservatives were suspected of involvement in terrorist plots: they were targeted on the basis of their political beliefs and activism, as part of a class the NSA refers to as "radicalizers."

I wonder if any members of this committee have ever advocated a position that the NSA, GCHQ, or even the intelligence services of an EU member state might attempt to construe as "radical"? If you were targeted on the basis of your political beliefs, would you know? If they sought to discredit you on the basis of your private communications, could you discover the culprit and prove it was them? What would be your recourse?

And you are parliamentarians. Try to imagine the impact of such activities against ordinary citizens without power, privilege, or resources. Are these activities necessary, proportionate, and an unquestionable matter of national security?

A few weeks ago we learned the GCHQ has hired scientists to study how to create divisions amongst activists and disfavored political groups, how they attempt to discredit and destroy private businesses, and how they knowingly plant false information to misdirect civil discourse (https://firstlook.org/theintercept/2014/02/24/jtrig-manipulation/).

To directly answer your question, yes, global surveillance capabilities are being used on a daily basis for the purpose of economic espionage. That a major goal of the US Intelligence Community is to produce economic intelligence is the worst kept secret in Washington.

In September, we learned the NSA had successfully targeted and compromised the world's major financial transaction facilitators, such as Visa and SWIFT, which released documents describe as providing "rich personal information," even data that "is not about our targets" (http://www.spiegel.de/international/world/spiegel-exclusive-nsa-spies-on-international-bank- transactions-a-922276.html). Again, these documents are authentic and unmodified - a fact the NSA itself has never once disputed.

In August, we learned the NSA had targeted Petrobras, an energy company (http://g1.globo.com/fantastico/noticia/2013/09/nsa-documents-show-united-states-spied- brazilian-oil-giant.html). It would be the first of a long list of US energy targets.

But we should be clear these activities are not unique to the NSA or GCHQ. Australia's DSD targeted Sri Mulyani Indrawati, a finance minister and Managing Director of the World Bank (http://www.theguardian.com/world/2013/nov/18/australia-tried-to-monitor-indonesian- presidents-phone). Report after report has revealed targeting of G-8 and G-20 summits. Mass surveillance capabilities have even been used against a climate change summit.

Recently, governments have shifted their talking points from claiming they only use mass surveillance for "national security" purposes to the more nebulous "valid foreign intelligence purposes." I suggest this committee consider that this rhetorical shift is a tacit acknowledgment by governments that they recognize they have crossed beyond the boundaries of justifiable activities. Every country believes its "foreign intelligence purposes" are "valid," but that does not make it so. If we are prepared to condemn the economic spying of our competitors, we must be prepared to do the same of our allies. Lasting peace is founded upon fundamental fairness.

The international community must agree to common standards of behavior, and jointly invest in the development of new technical standards to defend against mass surveillance. We rely on common systems, and the French will not be safe from mass surveillance until Americans, Argentines, and Chinese are as well.

The good news is that there are solutions. The weakness of mass surveillance is that it can very easily be made much more expensive through changes in technical standards: pervasive, end-to-end encryption can quickly make indiscriminate surveillance impossible on a cost- effective basis. The result is that governments are likely to fall back to traditional, targeted surveillance founded upon an individualized suspicion. Governments cannot risk the discovery of their exploits by simply throwing attacks at every "endpoint," or computer processor on the end of a network connection, in the world. Mass surveillance, passive surveillance, relies upon unencrypted or weakly encrypted communications at the global network level.

If there had been better independent and public oversight over the intelligence agencies, do you think this could have prevented this kind of mass surveillance? What conditions would need to be fulfilled, both nationally and internationally?

Yes, better oversight could have prevented the mistakes that brought us to this point, as could an understanding that defense is always more important than offense when it comes to matters of national intelligence. The intentional weakening of the common security standards upon which we all rely is an action taken against the public good.

The oversight of intelligence agencies should always be performed by opposition parties, as under the democratic model, they always have the most to lose under a surveillance state. Additionally, we need better whistleblower protections, and a new commitment to the importance of international asylum. These are important safeguards that protect our collective human rights when the laws of national governments have failed.

European governments, which have traditionally been champions of human rights, should not be intimidated out of standing for the right of asylum against political charges, of which espionage has always been the traditional example. Journalism is not a crime, it is the foundation of free and informed societies, and no nation should look to others to bear the burden of defending its rights.

Shadow Rapporteur Axel Voss MEP, EPP Group

Why did you choose to go public with your information?

Secret laws and secret courts cannot authorize unconstitutional activities by fiat, nor can classification be used to shield an unjustified and embarrassing violation of human rights from democratic accountability. If the mass surveillance of an innocent public is to occur, it should be authorized as the result of an informed debate with the consent of the public, under a framework of laws that the government invites civil society to challenge in open courts.

That our governments are even today unwilling to allow independent review of the secret policies enabling mass surveillance of innocents underlines governments' lack of faith that these programs are lawful, and this provides stronger testimony in favor of the rightfulness of my actions than any words I might write.

Did you exhaust all possibilities before taking the decision to go public?

Yes. I had reported these clearly problematic programs to more than ten distinct officials, none of whom took any action to address them. As an employee of a private company rather than a direct employee of the US government, I was not protected by US whistleblower laws, and I would not have been protected from retaliation and legal sanction for revealing classified information about lawbreaking in accordance with the recommended process.

It is important to remember that this is legal dilemma did not occur by mistake. US whistleblower reform laws were passed as recently as 2012, with the US Whistleblower Protection Enhancement Act, but they specifically chose to exclude Intelligence Agencies from being covered by the statute. President Obama also reformed a key executive Whistleblower regulation with his 2012 Presidential Policy Directive 19, but it exempted Intelligence Community contractors such as myself. The result was that individuals like me were left with no proper channels.

Are you aware that your revelations have the potential to put at risk lives of innocents and hamper efforts in the global fight against terrorism?

Actually, no specific evidence has ever been offered, by any government, that even a single life has been put at risk by the award-winning journalism this question attempts to implicate.

The ongoing revelations about unlawful and improper surveillance are the product of a partnership between the world's leading journalistic outfits and national governments, and if you can show one of the governments consulted on these stories chose not to impede demonstrably fatal information from being published, I invite you to do so. The front page of every newspaper in the world stands open to you.

Did the Russian secret service approach you?

Of course. Even the secret service of Andorra would have approached me, if they had had the chance: that's their job.

But I didn't take any documents with me from Hong Kong, and while I'm sure they were disappointed, it doesn't take long for an intelligence service to realize when they're out of luck. I was also accompanied at all times by an utterly fearless journalist with one of the biggest megaphones in the world, which is the equivalent of Kryptonite for spies. As a consequence, we spent the next 40 days trapped in an airport instead of sleeping on piles of money while waiting for the next parade. But we walked out with heads held high.

I would also add, for the record, that the United States government has repeatedly acknowledged that there is no evidence at all of any relationship between myself and the Russian intelligence service.

Who is currently financing your life?

I am.

Shadow Rapporteur Timothy Kirkhope MEP, ECR Group

You have stated previously that you want the intelligence agencies to be more accountable to citizens, however, why do you feel this accountability does not apply to you? Do you therefore, plan to return to the United States or Europe to face criminal charges and answer questions in an official capacity, and pursue the route as an official whistle-blower?

Respectfully, I remind you that accountability cannot exist without the due process of law, and even Deutsche Welle has written about the well-known gap in US law that deprived me of vital legal protections due to nothing more meaningful than my status as an employee of a private company rather than of the government directly (http://www.dw.de/us-whistleblower-laws-offer- no-protection/a-17391500). Surely no one on the committee believes that the measure of one's political rights should be determined by their employer.

Fortunately, we live in a global, interconnected world where, when national laws fail like this, our international laws provide for another level of accountability, and the asylum process provides a means of due process for individuals who might otherwise be wrongly deprived of it. In the face of the extraordinary campaign of persecution brought against me by my the United States government on account of my political beliefs, which I remind you included the grounding of the President of Bolivia's plane by EU Member States, an increasing number of national governments have agreed that a grant of political asylum is lawful and appropriate.

Polling of public opinion in Europe indicates I am not alone in hoping to see EU governments agree that blowing the whistle on serious wrongdoing should be a protected act.

Do you still plan to release more files, and have you disclosed or been asked to disclose any information regarding the content of these files to Chinese and Russian authorities or any names contained within them?

As stated previously, there are many other undisclosed programs that would impact EU citizens' rights, but I will leave the public interest determinations as to which of these may be safely disclosed to responsible journalists in coordination with government stakeholders. I have not disclosed any information to anyone other than those responsible journalists.

Thank you.

Discovering Ghost

For past blogging attempts I've used Wordpress (easy) and Jekyll (hackable) and neither were much fun. As a developer I want something that's easy and hackable and I'm hoping that the node based Ghost fits the bill.

I get to write Markdown (win) with a nice javascript-based preview window, and the posts save to an SQLite database.

Installation

wget https://ghost.org/zip/ghost-0.4.1.zip
unzip ghost-0.4.1.zip
cd ghost-0.4.1
brew update
brew install node
npm install --production
npm start

Very nice!

Hosting

I'd been hosting up to now on Github pages which to be honest has been a giant PITA (mainly because of issues with Octopress). I'm looking at moving to Heroku but in the interests of getting up and running quickly, I used buster to generate a static site from my local Ghost install.

sudo pip install buster
buster setup --gh-repo=git@github.com:xavriley/xavriley.github.com.git
buster generate --domain=http://localhost:2368
buster preview # browse localhost:9000 to check
buster add-domain xavierriley.co.uk
buster deploy

Except in my case buster deploy failed because of my existing Octopress blog in that repo. After taking a fresh clone to backup the old blog, I deployed by:

cd  static # where buster makes it's repo
git push -f origin master

which blows away the old repo and lets me deploy normally from then on.

For bonus points

To make the deploy process smoother, try running:

sudo sh -c 'echo "127.0.0.1 local.tryghost.org" >> /etc/hosts'

to be able to run buster generate with no extra args.

Conclusion

From discovering Ghost to deploying to Github pages took about 15 minutes and I didn't have any issue that held me up for more than about 10 seconds. Now I've been programming professionally for the last six years and I can honestly say that doesn't happen very often.

That said, there still seems to be a lack of good documentation
concerning proper Behaviour Driven Development (BDD) with PHP and it's a
problem that we came across recently at Kyan. I'm going to try and do a
walkthrough of setting up some simple fetaure testing using a default
install of the Prestashop ecommerce
framework. It's a fairly sizeable project that doesn't ship with test
coverage - let's fix that!

What you'll need

• A local install of Prestashop
• Firefox (just the standard consumer version, not Aurora or nightly)
• Selenium RC (2.25.0 at the time of writing)

• PHP 5.4 (just so we're all on the same page)

• Behat (a PHP testing framework)

• Mink (web acceptance testing for Behat)
• Composer (a PHP packaging tool - think Bundler for PHP)

1.) Install Prestashop

Prestashop has a good set of instructions so I won't duplicate them all
here. Suffice to say you need to set up a MySQL db and change a few
permissions.

2.) Install Composer

Make sure PHP is in your path. You can test this by firing up terminal
and typing

php -v  

If you're using MAMP, put something like
this at the bottom of your .bashrc

export PATH="/Applications/MAMP/bin/:$PATH"  

Double check the path to the MAMP binaries. It's different in different
versions of MAMP.

Now cd into your newly created prestashop folder and run the following
commands to install composer;

mkdir bin  
curl -s https://getcomposer.org/installer | php -- --install-dir=bin  

If you're installing using the default OSX apache and you get an error
about "detect_unicode = Off", you'll also need to
run the following;

sudo su  
echo "detect_unicode = Off" >> /private/etc/php.ini  
apachectl restart  
exit  

3.) Install Behat and Mink

This part is the reason I'm writing this blog post. There seems to be a
lack of clear information about how to get these up and running from
scratch. Hopefully this will be sorted out as time goes on but I would
suggest people submit their points about what works and doesn't work for
them.

To get started, make a file called composer.json in the root of the
Prestashop install and put in the following;

{
    "require": {
        "behat/behat":           "2.4@stable",
        "behat/mink-extension":  "*",
        "behat/mink-goutte-driver":     "*",
        "behat/mink-selenium2-driver":  "*"
    },
    "minimum-stability": "dev",
    "config": {
        "bin-dir": "bin"
    }
}

This is the equivalent of a Gemfile if you've used bundler and Ruby. So
to get everything up and running, run composer like so.

php bin/composer.phar install  

All being, well you should see the packages installing into a vendor
folder in the project root.

4.) Setting up Behat

> bin/behat --init                     
+d features - place your *.feature files here
+d features/bootstrap - place bootstrap scripts and static files here
+f features/bootstrap/FeatureContext.php - place your feature related code here

This bootstraps the files needed to run the tests. Let's go ahead and
write our first test. Usually in test driven development, we write the
tests before the code, but I'm advocating here that we can add tests to
and exisiting site to give us the confidence to refactor and add
features later on. Start editing /features/basket.feature and put in
the following;

Feature: Basket  
  In order to add a product to the basket
  As a website user
  I need to add the product to my basket
  And I should see the product in the basket 

  @javascript
  Scenario: Add a product to basket
    Given I am on a product page
    When I click "Add to cart"
    And I hover over "#shopping_cart a"
    Then I should see the product title 

This is an example of a Cucumber test (the natural language syntax is
referred to as Gherkin) which is a popular way of doing acceptance
testing in Rails and other frameworks. We proceed by running the test
like so;

> bin/behat
Feature: Basket  
  In order to add a product to the basket
  As a website user
  I need to add the product to my basket
  And I should see the product in the basket

  @javascript
  Scenario: Add a product to basket     # features/basket.feature:8
    Given I am on a product page
    When I click on "Add to cart"
    And I hover over "#shopping_cart a"
    Then I should see the product title

1 scenario (1 undefined)  
4 steps (4 undefined)  
0m0.039s

You can implement step definitions for undefined steps with these snippets:

    /**
     * @Given /^I am on a product page$/
     */
    public function iAmOnAProductPage()
    {
        throw new PendingException();
    }

    /**
     * @When /^I click "([^"]*)"$/
     */
    public function iClick($arg1)
    {
        throw new PendingException();
    }

    /**
     * @Given /^I hover over "([^"]*)"$/
     */
    public function iHoverOver($arg1)
    {
        throw new PendingException();
    }

    /**
     * @Then /^I should see the product title$/
     */
    public function iShouldSeeTheProductTitle()
    {
        throw new PendingException();
    }

This runs the test and tells you what to do next. Copy the snippets into
the features/bootstrap/FeatureContext.php file above the final
curly brace. After saving and running bin/behat again, the output should change to include

TODO: write pending definition  

5.) Enable Mink

This is where the documentation starts to part ways with the latest
versions of Mink and Behat. After struggling with the official run
through at http://mink.behat.org/ trying to
get Zombie working as the Javascript client, I stumbled across the Mink
example repository in the Behat official Github page here
https://github.com/Behat/MinkExtension-example

The key is swapping out BehatContext in the FeaturesContext file, to
make sure that it reads;

class FeatureContext extends Behat\MinkExtension\Context\MinkContext  

and then you need to configure Behat and Mink to be looking in the right
places and choosing the correct browser drivers. Make a file called
behat.yml in the root of the project with the following;

default:  
  context:
    class:  'FeatureContext'
  extensions:
    Behat\MinkExtension\Extension:
      base_url:  'http://tdd.prestashopexample.com/'
      goutte:    ~
      selenium2: ~

goutte seems to be a requirement as far as I can tell, but who knows...

Now when you run bin/behat again, you should see an error like this;

Curl error thrown for http POST to http://localhost:4444/wd/hub/session with params: {"desiredCapabilities":{"browserName":"firefox","version":"8","platform":"ANY","browserVersion":"8","browser":"firefox"},"requiredCapabilities":[]}  

Enter Selenium...

6.) Setting up selenium

This should be quite straightforward. First off, make sure you have the
standard version of Firefox installed. Download Selenium RC from the link
at the top of the page and move the .jar file to the vendor/ folder.
Then run the following command to start the Selenium server;

java -jar vendor/selenium-server-standalone-2.25.0.jar > /dev/null &  

You should see INFO: Launching a standalone server and you're good to
try the tests again by running bin/behat. This time you'll see Firefox
start up, flash up with the page content and close again. The first test
passed! Now let's implement the others.

7.) Finishing the steps

Change the methods in FeatureContext.php to look like this;

    /**
     * @Given /^I hover over "([^"]*)"$/
     */
    public function iHoverOver($arg1)
    {
        $this->getSession()->getPage()->find('css', $arg1)->mouseOver();
        $this->getSession()->wait(5000, "$('#cart_block_list').hasClass('expanded')");
    }

    /**
     * @Then /^I should see the product title$/
     */
    public function iShouldSeeTheProductTitle()
    {
        $product_title = $this->getSession()->getPage()->find('css', '.cart_block_product_name');
        $product_title == "iPod Nano";
    }

A few things to note here before we proceed - this isn't a very robust
test and could be improved. There's duplication all over the place which
could be refactored (but it's late and I've just got it working...). The
real problem is that it's assuming the title of the product as the
assertion in the last point. A better approch would be to call/mock the
product from Prestashop in the constructor and use that instead. Maybe
in part 2.

Also, it bugged me that the wait->(...) call which executes JS is
executed in the context of the session. It makes perfect sense in
hindsight as getPage is returning a sort of DOM object but it tripped
me up nonetheless.

Down to business - run bin/behat one last time and the output should
look like this;

> bin/behat                                                                                       1 ↵
Feature: Basket  
  In order to add a product to the basket
  As a website user
  I need to add the product to my basket
  And I should see the product in the basket

  @javascript
  Scenario: Add a product to basket     # features/basket.feature:8
    Given I am on a product page        # FeatureContext::iAmOnAProductPage()
    When I click on "Add to cart"       # FeatureContext::iClickOn()
    And I hover over "#shopping_cart a" # FeatureContext::iHoverOver()
    Then I should see the product title # FeatureContext::iShouldSeeTheProductTitle()

1 scenario (1 passed)  
4 steps (4 passed)  
0m4.056s  

Happy days!

Conclusion

There's a ton of PHP apps out there that would benefit from regression
tests like this. They're not that hard to write once you get going and
they give you an invaluable safety net and peace of mind when it comes
to refactoring and adding new features.

My motivation for writing this was that the current state of documentation
seems a bit patchy, despite the Behat and Mink projects being well established.
I suppose this is a call to arms to PHP devs to try this stuff out
and get their feedback into the loop.

I'd love to see more advanced posts dealing with the various drivers
that are available and also to hear what people think of
this one. I'll finish by saying I'm starting out on this trail so my advice
is only what I've managed to cobble together up to now. If there's
improvements or suggestions in the comments I'll happily fold them back in to
the main post. Happy testing...

When picking a solution it really pays to do some forward planning and in this case, Prestashop can’t scale up to the kind of multi-store functionality that this music department store needed. That said, it was a lot easier getting a Prestashop store off the ground (I tried both at the start) and it’s served us well. My only gripe is that Prestashop’s module and templating system isn’t flexible enough to make real changes without editing the core – and that screws up future updates. Not what you need for ecommerce purposes.

UPDATE
Since writing this, it's all change with Prestashop v1.5
They've had a basic overrides system in place for a while now.
Also the new version supports multistore but I haven't used it, yet...

Making the switch

Swapping out the products from PS to Mage is pretty straightforward – it’s just a case of a mysql query which maps them into the right fields. As with any Magento import the process is more or less similar;

• Add some dummy data into Magento, using all the fields you’re likely to use
• Export the data as a csv using System > Import/Export > Profiles
• Look at the file in your var/export folder and take a look at the headers

Tip for *nix users working with csv files – in terminal use the following command to save the first two lines to a file;

head -n 2 your_csv_file.csv > your_csv_file_head.csv  

That’s if your csv file is really big.

The problem with users

Now the above works fine for most things, but the problem with users lies in the different authentication that both shops use. Magento uses MD5 with a salt on the end, Prestashop uses a ‘Cookie Key’ prefix to the customer password, which is then MD5 encrypted.
Anyone who’s looked into MD5 knows that this is a pig – you can’t reverse an MD5 into plain text, therefore you can’t get the original password strings in order to re-encode them (which is actually a good thing…).

How to fix it

Clearly its impossible to convert from one MD5 hash to another so we have to try something different. Make the following file;

(magentoroot)/app/code/local/Mage/Customer/Model/Customer.php

and in it put the following;

class Mage_Customer_Model_Customer extends Mage_Core_Model_Abstract  
{
/**
* Authenticate customer
*
* @param string $login
* @param string $password
* @return true
* @throws Exception
*/
public function authenticate($login, $password)  
{

$this->loadByEmail($login);
if ($this->getConfirmation() && $this->isConfirmationRequired()) {  
throw Mage::exception('Mage_Core', Mage::helper('customer')->__('This account is not confirmed.'),  
self::EXCEPTION_EMAIL_NOT_CONFIRMED  
);
}
if (!$this->validatePassword($password) && !$this->validatePassword('hKvthisisyourgibberishcookiestringfromprestashopCM'.$password)) {  
throw Mage::exception('Mage_Core', Mage::helper('customer')->__('Invalid login or password.'),  
self::EXCEPTION_INVALID_EMAIL_OR_PASSWORD  
);
}
Mage::dispatchEvent('customer_customer_authenticated', array(  
'model' => $this,  
'password' => $password,  
));
return true;  
}

//end class
}
}

Notice the random string prepended to the password variable? That’s your cookie string which you’ll find in your Prestashop install here;

(prestshop root)/config/settings.inc.php

It’s the line beginning

define('_COOKIE_KEY_', 'ThisIsTheBitYouWant...'  

Believe it or not that’s all you need to do. Import your MD5 hashes from Prestashop straight into Magento and it’ll authenticate them as Prestashop would do. I’ll go through the mysql import in another post.