offers hundreds of practice questions and video explanations. Go there now.
Sign up or log in to Magoosh GRE Prep.

GRE Math Challenge: Polygon Problem

Over the past few weeks, I have covered several tricky math concepts. Below is a problem that incorporates many of these lessons into one problem.  If you are up to it, definitely take a stab at this tough, time-consuming problem. If you want a little primer (or already gave it a crack, but are unsure how to proceed), have a look here at combinations and permutations.

How many unique quadrilaterals can be inscribed in the vertices of a nonagon (a 9-sided figure), if points A and B, two vertices in the nonagon, cannot make up the same quadrilateral? 

(A)  126

(B)  105

(C)  96

(D)  65

(E)  21

How to Crack It

We know that any four out of nine points can make a quadrilateral. If we approach this problem as a combinations question, then we want to figure out how many ways we can select 4 out of 9 points. This will yield the total. From this, we want to subtract those points that contain both the letters A and B.

First, though, let’s find the total number of quadrilaterals: (9 x 8 x 7 x 6)/(4 x 3 x 2 x 1) = 3 x 6 x 7 = 126.

Next, and here’s the tricky part, we have to figure out how many quadrilaterals contain both points A and B. You could write down those points: ABCD, ABCE, etc. But that, of course, would take forever, and give rise to writer’s cramp faster than it will yield an answer.

Instead, let’s look at this logically: two points are already determined, A and B. There are 7 remaining points that could occupy the two other vertices of the quadrilateral (we can think of it as ABXX – we have to choose two out of the seven letters to fit in the XXs). Therefore, using combinations, we get (7 x 6)/(2 x 1) = 21.

That is, there are 21 quadrilaterals that contain points A and B. We have to discount these if we want to answer the question. Therefore, we take the total, 126 – 21 = 105 (B).


This is obviously a very difficult problem: it is conceptually tough, and you have to use embedded combinations (that just sounds scary). But, hopefully, after watching the videos and/or reading the posts on the various concepts, you can combine the different elements necessary to solve this problem.

Again, as with all challenge problems, don’t despair, unless you positively have to break 700.



By the way, students who use Magoosh GRE improve their scores by an average of 8 points on the new scale (150 points on the old scale.) Click here to learn more.

8 Responses to GRE Math Challenge: Polygon Problem

  1. Annina July 5, 2014 at 2:49 am #

    I’m confused by the premise of this question. What is the meaning of a polygon being “inscribed” within an other? I thought it meant ‘contained within the larger polygon, with the vertices of the inner polygon touching the outer polygon, and none of their lines crossing’. So all the vertices of the quadrilateral would have to touch the lines or vertices nonagon.

    However, you seem to be assuming that the vertices of the quadrilateral have to specifically touch the VERTICES of the nonagon – they cannot be located along the edges of the nonagon. With this assumption, your solution makes perfect sense to me. With the second case, where the quadrilateral’s vertices are allowed to be located along an edge of the nonagon, which my definition of “inscribed” led me to believe was the situation, there would be infinitely many possile quadrilaterals (as any of the quadrilateral’s vertices could be moved an infinitessimally small amount along an edge of the nonagon to create a new quadrilateral).

    Do I have the definition of “inscribed” wrong?

    Also, you use the phrase “points A and B, two points in the nonagon” to signify “points A and B, two vertices of the nonagon”, as opposed to any points contained in the lines or area of the nonagon, of which there would be infinitely many. Is this a question copied from the ETS, i.e. do I have to learn to deal with this type of ambiguity in maths questions? When a question mentions points in a polygon, do I have to just assume they mean vertices?

    • Chris Lele
      Chris Lele July 9, 2014 at 5:18 pm #

      Hi Annina,

      It seems that I made a mistake. You are right: the question should say VERTICES of the nonagon. This is a question that I put up for the blog, so it is not an ETS question, nor a perfect question. Meaning that readers will sometimes point out something that is off. The strange thing with this question is it has been up for nearly three years and you were the first one to point out the ambiguity. Again, my apologies. I will change the text accordingly.

  2. Aditya Swarup August 22, 2013 at 12:49 am #

    Milon: Jonah’s approach is correct but explanation is wrong. Actually the appropriate explanation goes like this:
    Include A(and exclude B):That means vertex A is fixed, remaining 7 vertices of nonagon can be chosen as 3 vertices of the quadrilateral in 7 choose 3 ways(Jonah’s method works out because 7 choose 3 is equal to 7 choose 4 based on the combinations formula)

    Include B(and exclude A):Same explanation as above resulting in 7 choose 3

    Without A and B: 7 choose 4

    Result= 2*7C3+7C4=105, which works out equal to 3*7C4 since 7C4=7C3

    • Chris Lele
      Chris Lele August 23, 2013 at 1:46 pm #


      Thanks for catching that–it is subtle. But you’re right–once ‘A’ is fixed you are only choosing 3 of 7, not 4. Had the question had a decagon, then the Jonah’s logic would have turned out incorrectly (8C4 vs the correct 8C3).

      This is quite an interesting question type. I’m tempted to come up with another such question!

  3. jonah August 3, 2012 at 3:43 pm #

    Hey Chris,

    I was curious to get your thoughts on solving the problem just using 3*(7choose4) = 105.

    My thought process was:

    Include A: 8 left over but can’t include B, so 7choose 4

    Include B: 8 left over but can’t include A, so 7choose 4

    Don’t include either: 7 left over, so 7 choose4.

    I guess I wanted to know if this method could get me into trouble on similar problems but with a different twist. Or are the two methods basically interchangeable in terms of applicability on actual GRE-style questions?

    Thanks so much!

    • Chris Lele
      Chris August 6, 2012 at 1:07 pm #

      Hi Jonah,

      That way is totally valid as well. It is also the way I approached the question just now, after doing it with fresh eyes (meaning I hadn’t seen this problem in months). So, no: thinking of a problem in a different manner will not get you into trouble :).

      • Milon August 11, 2013 at 4:10 pm #

        How could Jonah’s method valid? When you include A but B, the combination is 8C4, NOT 7C4. In the same way, if you include B but A yields 8C4. The way you solve the problem is appropriate but Jonah’s method seems wrong. Can you evaluate Jonah’s solution once again please?

    • ram May 15, 2013 at 12:06 am #

      nice approach jonah..

Magoosh blog comment policy: To create the best experience for our readers, we will only approve comments that are relevant to the article, general enough to be helpful to other students, concise, and well-written! 😄 Due to the high volume of comments across all of our blogs, we cannot promise that all comments will receive responses from our instructors.

We highly encourage students to help each other out and respond to other students' comments if you can!

If you are a Premium Magoosh student and would like more personalized service from our instructors, you can use the Help tab on the Magoosh dashboard. Thanks!