>"It showed that a top employee averaged 281 billion tokens per month (~$1.4million in API costs). To compare, Alfred, which is running as my personal chief of staff averages around 2-4 billion tokens per month."
Using the same cost Meta referenced, you're spending $10-20k per month on your agent. I assume that's not the case and you have some optimizations to bring that number way down?
yep, well two things. i'm using subsidized plans now (currently on codex, but i tested kimi code and minimax too).
i'd say the real API cost would average around $6-8k if i default to frontier models but:
i'm working on optimizations to bring that number down even further.
minimax m2.7 is has very competitive api pricing (~1/20th vs opus) and loads of the work that actually requires an llm is offloaded to that.
the most token intensive work is signal extraction from raw data (calendar, chat, email, slack, etc) and distillation. both can be optimized to minimize actual llm use
whoever wants my personal data already has it and there's not much i can do about it so generally i don't think about it too much. using openai, anthropic or google isn't "safer" either. the only safe option is to not be online
I appreciate the honesty. I'm not there yet but if you are right about token costs ballooning eventually I'll likely have to re-underwrite just how much I value that specific slice of privacy that I'm currently willing to pay a little extra for.
>"It showed that a top employee averaged 281 billion tokens per month (~$1.4million in API costs). To compare, Alfred, which is running as my personal chief of staff averages around 2-4 billion tokens per month."
Using the same cost Meta referenced, you're spending $10-20k per month on your agent. I assume that's not the case and you have some optimizations to bring that number way down?
yep, well two things. i'm using subsidized plans now (currently on codex, but i tested kimi code and minimax too).
i'd say the real API cost would average around $6-8k if i default to frontier models but:
i'm working on optimizations to bring that number down even further.
minimax m2.7 is has very competitive api pricing (~1/20th vs opus) and loads of the work that actually requires an llm is offloaded to that.
the most token intensive work is signal extraction from raw data (calendar, chat, email, slack, etc) and distillation. both can be optimized to minimize actual llm use
You don't have any concerns using models from Chinese companies with your personal data?
whoever wants my personal data already has it and there's not much i can do about it so generally i don't think about it too much. using openai, anthropic or google isn't "safer" either. the only safe option is to not be online
I appreciate the honesty. I'm not there yet but if you are right about token costs ballooning eventually I'll likely have to re-underwrite just how much I value that specific slice of privacy that I'm currently willing to pay a little extra for.
i'll do an analysis of token costs vs performance, comparing different models, providers vs local models too.
generally speaking the "near opus level fully local agent" pricetag is at around $100k now to be actually useful.
if there was an option to lease such hardware at a reasonable monthly cost i would do so in a heartbeat.