P.S.: This quick demo shows what you wanted from your first post. The 16th tuplets have the 3 hidden, as your demo did, but I think they should show, personally.

Favorite Forum quote: "Please, everybody, IGNORE THE TROLL!"

Post Edited (Zuill) : 2/25/2013 1:09:11 AM (GMT-6)

Image Attachment : Nested Tuplet Demo.png   10KB (image/x-png) This image has been viewed 659 time(s).

One must also point out then that one needs to be in Replace mode to begin with, as denoted by a subtle change in the cursor.

As I said I would do, please find included four ways that I was able to form the first complex nested tuplet given by the OP, two using Simple entry, and two using Speedy entry.
These are not the only ways to form the tuplet.  Others have posted their methods.  Later, I will post methods I found to form the second example of a complex tupet posted by the OP.

This post is long, and for that my apologies, but I tried to be detailed and complete, so that a novice could follow the steps.  I could find nothing in the documentation that explained any steps necessary to form complex nested tuplets, except to say that it could be done.  I don't consider these methods straightforward.  They only become so with familiarity.  Being a physicist/mathematician by occupation (retired), I would find the derivation of Maxwell's equations of classical electrodynamics to be "straightforward" after completing a course of study, but I dare say the very looks of them as partial differential equations would be disconcerting and incomprehensible to the uninitiated, as they were to me when I first saw them.  So, for what it is worth, here are my methods.  I hope they are clear and free of errors.  If you find any errors, please point them out.

Simple Entry - Outside tuplet 1st, then inside tuplets

1) Choose the 8th note
2) Click in an 8th note in an empty 4/4 measure
3) Press ALT-9.  Make sure 3 eighths in the space of 2 eighths is chosen.  OK.
4) Overstrike resulting 2 eighth rests with 2 eighth notes.  This completes the outside tuplet.
5) Select the Tuplet Tool (not to be confused with the tuplet modifier in the Simple Entry palette).  This will fill out the measure with rests, provided Simple Options are set to do so.
6) Double click the 1st eighth note to bring up the "Tuplet Definition" dialog box.  Make sure 3 16ths in the space of 1 eighth is chosen. OK.
7) Choose the Simple Entry Tool again.  CTRL-click the first eighth note to select it.  Press ALT-3 (numpad) to change it to a 16th note.
8) Choose the 16th note (with no modifiers).  Zoom in to at least 200% to make the next steps easier
9) Carefully position the Simple Entry 16th note cursor to between the just changed 16th note and the next 8th note to its right.  Click in the 2nd of the 16th notes of the first inside tuplet.
10) Carefully position the cursor to between the just entered 16th note and the eighth note to its right.  Click in the 3rd of the 16th notes of the first inside tuplet.
11) Repeat steps 5 thru 12 except in step 6) choose the last of the eighth notes of the outside tuplet to double click on, and in steps 11) and 12) position the cursor to between the 16th note and the quarter note rest to its right.
12) And there you have it.  You may want to remove the number "3" on the two 16th note inside tuplets.  This is done with the Tuplet Tool by clicking on the 1st of the 16th notes in the group, then right clicking on the tuplet handle and choosing to "Edit Tuplet Definition...".  When you click on the 1st 16th note, observe that two handles appear, because that note is the 1st note in 2 tuplets, the 16th note inside tuplet as well as the 8th note outside tuplet.  Make sure to right click on the left most handle, the one attached to the 16th note tuplet.  In the "Appearance" Grouping, choose "Nothing" in the dropdown for "Number".  Repeat for the 2nd 16th note tuplet.

Simple Entry - Inside tuplets 1st, then outside tuplet.  (This is actually easier than the other way.)

1) Choose the 16th note
2) Click in a 16th note in an empty 4/4 measure
3) Press ALT-9.  Make sure 3 16ths in the space of 1 eighth is choosen.  OK.
4) Overstrike the resulting 2 16th note rests with 16th notes.  This completes one inside tuplet.
5) Choose the 8th note.
6) Click in an 8th note.
7) Repeat steps 1) thru 4).  This will complete the 2nd inside tuplet.
8) Select the Tuplet Tool.  This will fill out the measure with rests provided the Simple Options are set to do so.
9) Double click on 1st 16th note in the first tuplet to bring up the "Tuplet Definition" dialog box.  Make sure 3 8ths in the space of 2 eights is chosen.  OK.
10) And there you have it.  As before, you may want to remove the inside tuplet numbers.  Also the rests that fill out the measure may need some adjusting to get what you want.  Just delete them all within Simple entry and exit the measure.  Finale will refill the measure with a quarter note and a half note rest.  MIDI quantization settings will affect this too.

Speedy Entry - Outside tuplet 1st, then inside tuplets

1) Click in an empty 4/4 measure.  Position the cursor on the note you want using the up/down arrows
2) Press CTRL-1 to bring up the "Tuplet Definition" dialog box.  Make sure 3 eighths in the space of 2 eighths is chosen. OK.
3) Press numpad 4 (8th note) three times to enter the outside tuplet.
4) Select the Tuplet Tool.  This will fill out the measure with rests provided Speedy Options is set to do so.
5) Double click on the 1st eighth note of the outside tuplet.  This brings up the "Tuplet Definition" dialog box.  Make sure 3 16ths in the space of 1 eighth is chosen.  OK.
6) Double click on the last eighth note of the outside tuplet.  This brings up the "Tuplet Definition" dialog box again.  Make sure 3 16ths in the space of 1 eighth is chosen.  OK.  The two inside tuplets have now been "defined".
7) Select Speedy entry and click in the measure to bring up the edit frame again.  Position the cursor over the 1st eighth note of the outside tuplet.  Speedy entry is in the "Replace" mode, so if you press a numpad note value, like 3 for a 16th note, the note under the cursor, the 8th note in this case, will be replaced with a 16th note.
8) In order to complete the 1st 16th note inside tuplet with 2 more 16th notes, you have to put Speedy entry into "Insert" mode.  Since my keyboard does not have an Insert key (Ins) except in the numpad, this is easiest to do with the Speedy Menu.  That way you don't have to keep toggling "Num Lock" on an off to use the Ins key (0), an error prone procedure for sure. "Arrow heads" will be added to the top and bottom of the vertical cursor bar to indicate that you are now in "Insert" mode.  Press the numpad 3 (16th note) twice to complete the tuplet.
9) Place Speedy entry back into "Replace" mode using the Speedy menu.  Position the cursor over the last 8th note in the outside tuplet, and replace it with a 16th note by pressing numpad 3.
10) Go back into "Insert" mode and press numpad 3 (16th) twice to complete the 2nd inside tuplet.
11) And there you have it.  As before, you may want to remove the inside tuplet numbers.

Speedy Entry - Inside tuplets 1st, then outside tuplet.  (This is actually easier than the other way.)

1) Click in an empty 4/4 measure.  Position the cursor on the note you want using the up/down arrows.
2) Press CTRL-1 to bring up the "Tuplet Definition" dialog box.  Make sure 3 16ths in the space of 1 eighth is choosen. OK.
3) Press numpad 3 (16th note) three times to complete the 1st inside tuplet.
4) Press numpad 4 once to enter an eighth note.
5) Repeat steps 2) and 3) to complete the last inside tuplet.
6) Select the Tuplet Tool.  This will fill out the measure with rests provided the Speedy Options are set to do so.
7) Double click on 1st 16th note in the first tuplet to bring up the "Tuplet Definition" dialog box.  Make sure 3 8ths in the space of 2 eights is chosen.  OK.
8) And there you have it.  As before, you may want to remove the inside tuplet numbers.  Also the rests that fill out the measure may need some adjusting to get what you want.  Just delete them all within Speedy entry and exit the measure.  Finale will refill the measure with a quarter note and a half note rest.  MIDI quantization settings will affect this too.

Dell XPS 600, GenuineIntel Intel(R) Pentium(R) D CPU 3.00GHz [Intel64 Family 15 Model 4 Stepping 4] (2 processors)
8GB Ram
HT Omega Striker 7.1
MSI N430GT 2GB GPU
1TB x 4 internal HD's
Microsoft Windows 7 Ultimate x64 Edition, (06.01.7600.00)
Finale 2012c.r13

Post Edited (Charles Lawrence) : 2/26/2013 12:53:30 PM (GMT-6)