You would have to "hack" around it kind of.
What i mean is this, you put the category url template variable inside the class name to target more preceisly.
e.g class="category-btn {$category_url}"
Then you can target the button globally to have a default in case you forget some categories, plus individually
CSS:
.category-btn {
/* default color */
color: #ff0000;
}
.category-btn.My-Awesome-Category-Url {
/* default button overwrite for category 1 */
color: #000000;
}