The first time I flew this rocket, I discovered a couple flaws in my theories. First, I'd made the socket in the transition where the upper stage motor plugged in a little too snug. The gap staging worked fine, but with the vents, the upper stage wasn't able to build up enough pressure to free itsefl from the booster and burned while still attached. Then the sustainer motor ejected from the upper stage instead of fully deplaying the nose cone into a tumble configuration; the booster section was stable due to its large fins and the heavy transition/motor unit at the forward end, and streamlined in while the upper stage core sampled.
The fixes for the next one will be: extend the BT-5 for the booster mount all the way through the transition to ensure a proper fit at the staging joint, move the gap staging vents to behind the transition, and slice the transition lengthwise to allow it to eject and destabilize the booster for recovery. I doubt there's anything I can do to get away from friction fitting the upper stage motor, but that should be less of a problem without the staging difficultires.