Framing the problem

Questionnaires are part of the job. They’re part of how our industry works. Organisations need to make sure that their vendors are trustworthy enough to do business with, and questionnaires are the industry agreed-upon way to do so, alongside compliance frameworks obviously.

The problem is that questionnaires are often too generic. Customising questionnaires for each potential vendor introduces significant overhead for a third-party vendor risk assessment team. Historically, teams devised a standardised questionnaire based on common compliance frameworks, and sent that to all sorts of vendors. Fancy questionnaires came with some conditional logic to tailor it to specific use cases. Some organisations converged towards extremely detailed standard questionnaires like CSA CAIQ and SIG.

In the 2010s, security questionnaires got so repetitive and devoid of any originality that tools like Loopio adapted their RFP-answering offerings for this use case. Today, LLMs are making short work of security questionnaires, significantly unburdening go-to-market & security teams (thankfully, might I add).

Now here’s a question: if a machine is able to answer a security questionnaire, what value did that questionnaire really provide?

What questionnaires typically miss

The above question is clearly tongue-in-cheek. But it does highlight how the traditional questionnaire model was never really about security. It’s about the appearance of due diligence.

You, however, are a serious security professional. You don’t want to check a box and call it a day. You want to ask the hard-hitting questions. So here are a few you can ask next time you send a SaaS vendor a questionnaire. They will hopefully refocus your efforts on the actual assessment of security risks.

Can your staff impersonate users’ accounts?

It’s an open secret in the SaaS world: vendor staff can often log into your workspace as if they were you. It’s called impersonation. This is frequently a deliberate product decision, because it makes customer support teams’ lives easier, and debugging simpler for engineers too. The implication, though, is significant: any vendor employee with impersonation permissions could read your data, trigger actions on your behalf, and leave no obvious trace in your own audit logs.

Beyond whether it’s possible, the interesting follow-up questions are: How many staff can do this? Can you turn it off? Is impersonation read-only or can staff make changes on your account? Are sensitive values hidden during impersonation sessions?

A vendor who can give you a precise headcount of impersonation-capable employees, and who offers a customer-controlled toggle to disable it, has clearly thought about this as a security property. A vendor who answers vaguely is telling you something important about how they think about your data.

Describe who has read access to live production databases

Production database access is a genuinely difficult topic, and I say that with sympathy. Some bugs can only be reproduced in production, and engineers under pressure to fix something will take the path of least resistance. Over time, that path becomes a habit: access gets granted, never revoked, and suddenly a significant slice of the engineering org has standing read access to live customer data.

Ask the vendor to describe who currently has this access. Not who should have it, but who actually does. Request actual numbers. If it’s all engineers, that vendor is not very serious about protecting their customers’ data.

Where is production data copied?

Most SaaS vendors run a data analytics function of some kind — product analytics, revenue reporting, customer success dashboards. That work needs product usage data, and the path of least resistance is to pipe a copy of production into a data warehouse like Snowflake or BigQuery. This happens quietly, routinely, and often without any security review: a data engineer writes a sync job, and suddenly production data (also known as your data) lives in a second place, with entirely separate access controls, retention policies (if any), and an expanded attack surface.

The questions worth asking: how does the vendor manage the business intelligence teams’ access to production data? What datastores do they have? What data is in there? Are sensitive values scrubbed before data lands in the warehouse? A vendor without straightforward answers to these questions likely hasn’t modelled this as a risk surface at all.

Do you support on-demand & auditable privilege escalation?

Standing privileges are the easy default for organisations. Give an employee access once, and forget about it. For critical assets though, on-demand escalation is the harder, much more secure alternative. With on-demand privilege escalation, access to sensitive systems is not granted permanently, but requested at the time it’s needed, approved (or auto-approved based on policy), time-limited, and fully logged.

Such systems require solid IAM fundamentals and add operational complexity, which is exactly why most organisations don’t bother. But the security benefit is significant: an engineer’s compromised credentials, or a malicious insider, can only do damage within the narrow window of an active escalation session, not indefinitely.

On average, how often is 2FA requested for all employees? What about highly privileged employees?

The industry has broadly won the battle of getting 2FA enabled. Most compliance frameworks require it, and IDPs allow admins to enforce it. So most vendors can confidently tick that box today.

But “box ticked” doesn’t mean “2FA is effective at reducing authentication compromise risk”. If users get a 2FA challenge once every 30 days, and 2FA challenges are totally oblivious to context (eg. location, service sensitivity etc.), I’d argue it’s not very effective at mitigating authentication risks.

The more meaningful signal is challenge frequency: how often are employees actually prompted to re-authenticate? And crucially, is that number different for employees with access to production systems or customer data? A vendor who has thought seriously about this will have tiered session policies (shorter lifetimes and more frequent re-challenges for privileged roles) and will be able to tell you what those numbers look like, with a distribution graph for example.

What percentage of employees have onboarded a WebAuthn-compatible second factor?

Still on the topic of 2FA: TOTP works, but it’s phishable. It’s obviously miles better than nothing at all. But today we have much nicer, better things, like WebAuthn! Think passkeys & yubikeys.

If the vendor you’re eyeing is stuck on OTPs, it tells you something about their security posture.

Going further, the interesting question isn’t whether the vendor supports WebAuthn for their workforce. The question is how many employees have actually enrolled a compatible factor, and how often are they getting a 2FA challenge.

How do you encrypt sensitive values within your databases?

Most SaaS products have integrations with other services. Those integrations need credentials (eg. API keys, OAuth tokens) and those credentials are usually stored in the app databases.

Database disk-level encryption (ie the checkbox that turns that Vanta alert green) doesn’t help here: it protects against someone walking out with a hard drive, not against a compromised application or a rogue engineer running a query. What matters is whether your vendor is storing sensitive values encrypted at the column or field level, with keys that are separate from the data itself. Otherwise, it means any employee with database read permissions has access to your secrets.

Describe your security team’s size, budget and reporting line

Some questionnaires ask vendors to describe security roles and responsibilities. That’s fine, but it’s also easy to make bold and impressive claims on paper. The cold hard data is much more revealing.

Security headcount relative to the size of the organisation tells you whether security is a real function or a single person wearing many hats. You can also focus on the total security budget (as a percentage of total expenditure), rather than just team size. Narratives are cheap, but budget line items reflect the real priorities.

The reporting line also tells you something about organisational priorities. A CISO who reports to the CEO signals that security has executive weight. If they’re buried under the CFO or CTO, they may find it harder to push back on product or cost decisions.

Calibrating expectations

The questions we’re suggesting here require concrete data points. An LLM can generate answers for them, but only if the vendor actually did the prior work of aggregating the data. And if those answers are hallucinated by an LLM, you can call the bluff and probe with follow-up questions that only someone with real access to the data could answer.

Now, should you interrogate every vendor with the questions above? No, of course not. For most situations, your standard questionnaire is probably fine.

Use these questions for when you actually care that your vendor is not playing fast and loose with your data.

Not all vendors will be able to answer everything here fluently, and that’s not automatically disqualifying. A small startup with strong fundamentals may not yet have on-demand privilege escalation, but can explain their roadmap and their current compensating controls. What you’re really trying to determine is whether a vendor has thought about these things.