A few weeks ago, I attended a class in which Geoff Pullum did just this. He looked at flaws in Chomsky's coordination rule (26) on p. 36:
If S1 and S2 are grammatical sentences, and S1 differs from S2 only in that X appears in S1 where Y appears in S2 (i.e., S1 = . . .X . . . and S2 = . . .Y . . .), and X and Y are constituents of the same type in S1 and S2, respectively, then S3 is a sentence, where S3 is the result of replacing X by X + and +Y in S1 (i.e., S3 = . . .X + and +Y . . .).(This rule is reformulated as 22 on p. 113, which I won't bother reproducing here.)
Pullum pointed out that these rules are not even meaning equivalent: (26) requires the coordinated elements to be different while 22 doesn't. He also showed that they force each pair of coordinated constituents to be joined by and, for example: painter and carpenter and bricklayer. There's no way to produce painter, carpenter, and bricklayer. It's not like nobody had noticed these kinds of coordination. Reed and Kellogg, for instance include it in a 19th century school grammar.
The rules also overgenerate. That is, they produce strings that are ungrammatical:
- *the and the apple
- *the apple and and or the orange.
Never having read Syntactic Structures, I was intrigued enough to pick it up and read it. It was actually much easier than I'd expected. And in the reading, I found other rather obvious problems. Again, this may be with the benefit of 50 years of advances, but none of these things were unknown at the time.
Take the following rule (33) from p. 41.
Moving on, we get to the passive transformation (34) on p. 43. Chomsky broadly claims that, "for every sentence NP1 –V–NP2 we can have a corresponding sentence NP2–is + Ven – by NP1" (p. 43). This roughly includes all matrix clauses with transitive verbs, but not prepositional passives like it wasn't spoken of. Again, prepositional passives were well known at the time (e.g., Van der Gaaf 1930), but perhaps this oversight can be forgiven, since the book is meant as a sketch.
What's much worse is that it takes sentences like She left last week and turns them into *last week was left. Woops! It also produces *A butterfly was become by the caterpillar. Woops! The problem here is that the rule roughly picks out matrix clauses with transitive verbs. But Chomsky employs categories like NP and VP instead of syntactic functions like object, complement, and adjunct. Nowhere does he actually say the verb has to be transitive, just that it has to be followed by an NP. Unfortunately, this also captures NP predicative complements (became a butterfly) and NP adjuncts (left last week).
If (34) did specify objects, the term would have to be defined, and perhaps this is what Chomsky's trying to avoid. But even if he did, (34) would still produce sentences like *yourself is deceived by you. Again, this was a known issue.
There's a good deal more, but this gives you an idea of how "rigorous" Chomsky's "attempt to construct a grammar" with "carefully axiomatized and consistently detailed" rules (Lees, 1957) actually is.
Chomsky, Noam. 1957. Syntactic structures. The Hague: Mouton.
Lees, Robert B. 1957. Syntactic structures by Noam Chomsky. Language 33(3). 375–408.
Van der Gaaf, W. 1930. The passive of a verb accompanied by a preposition. English Studies 12(1–6). 1–24. doi:10.1080/00138383008596560.