These piles of dirt are accumulated frass and silk webbing from a caterpillar, [Pyralidae: Penthesilea
difficilis], tunneling inside the stem (photo with flower and frass by Stephen Brown, UF|IFAS Extension Lee Co.)There isn’t too much one can do with pesticides since the plants are used to attract butterflies and bees. About the only thing to do is be diligent and prune the affected stems before the caterpillar (in the banner picture) changes into a moth (see image below) and procreates more offspring that will damage more stems. Cut the stems into 2 inch long fragments to kill larvae. I’m not sure if the native porterweed is affected. And its doubtful that the porterweed will die from the damage. Its more of an aesthetic distraction.
See info on the different porterweed species here: