Cloud Hosting

(Editors Note: This article is the fifth and final in a series that is diving into the foundational barriers to the broad integration of AI in the IC culture, budget, acquisition, risk, and oversight. This article considers a modified approach to congressional oversight of the IC. The authors full report examining all of the topics in this series in greater depth is available here.)

Throughout this series, I have explored the most pressing foundational issues impacting the Intelligence Communitys (IC) ability to meet the urgency of this moment in the global artificial intelligence (AI) race. The current bipartisan support for taking bold action to drive national security use of AI is key to the ICs success. That support must propel change in the priority areas I have already identified: modernizing the ICs budget and acquisition processes and enabling a risk-tolerant culture with a new IC AI risk assessment framework that helps IC officers navigate the uncertainty that necessarily accompanies technological innovation. There is one other area, however, that cannot be ignored if the IC is to keep pace with our nations adversaries and provide policymakers with accurate, timely, and impactful insights: congressional oversight.

Congressional oversight of the IC is critical. Congress is the eyes and the ears of the American people. Among other things, it is charged with evaluating IC program performance, and ensuring the IC is efficiently and effectively spending taxpayer dollars and properly executing national security activities consistent with statutory requirements and legislative intent.

But intelligence oversight is complicated and has not sufficiently evolved with the times. When it comes to assessing progress of IC programs, standard oversight processes typically track defined, pre-determined requirements, cost, and timelines. These metrics have worked reasonably well for large programs like the acquisition of satellites and buildings, for which there is a clear beginning, middle, and end, with easily identifiable milestones and a definite budget. However, AI is different; its development moves back and forth across a spectrum of activities often without discrete steps, and failure is a necessary part of the process as the technology evolves and matures. Traditional metrics are, therefore, less effective for AI, as the value (or lack thereof) of certain milestones may only become clear partway through the development process and desired end-states may shift.

The IC has four primary congressional oversight committees. In addition to the House Permanent Select Committee on Intelligence (HPSCI) and the Senate Select Committee on Intelligence (SSCI), which have oversight jurisdiction over the IC, the House Appropriations Committee Defense Subcommittee (HAC-D) and the Senate Appropriations Committee Defense Subcommittee (SAC-D) provide the ICs money. These four committees (hereinafter collectively Committees) must consider a more adaptive approach to oversight, measuring progress and failure through metrics that are less rigid and better tailored for AI and other emerging technologies. In doing so, the Committees may lose a measure of certainty that impacts their most powerful lever fiscal control over the IC. For that reason, the Committees and the IC must simultaneously build a greater degree of trust, transparency, and ultimately partnership.

Much like AI itself, congressional oversight of AI activities must evolve and adapt to the world of emerging technology. While there are a variety of rules that govern Congress oversight responsibilities, Congress has considerable latitude and discretion in the execution of that oversight, including how they measure executive branch progress. To improve IC oversight engagements, Congress and the IC must start with a shared strategic vision for what a successful AI project looks like and create an approach to oversight that is tailored to achieve this goal.

Current measures and metrics often focus on ensuring projects stay on track in terms of cost and schedule; there are well-defined outputs, such as number of tools built, and static timelines for delivery. Such demonstrable deliverables are objective, consistent, and easy to measure, but they are ill-suited to AI, the underlying technology for which is still evolving. To take full advantage of AIs emerging possibilities, the IC must have the ability to test, adjust, and pivot as new algorithms and capabilities are developed and applied to different problem sets.

Recognizing that detailed requirements and program schedules are not well-suited to measuring the success of software, which is the core of AI, the Defense Department is already considering changes to related oversight. Research by Google and others indicates that metrics aligned with DevSecOps, the industry best practice for rapid, secure software development, can better predict the performance of software teams. DevSecOps seeks to unify software development (Dev), security (Sec), and operations (Ops). Its metrics allow teams to focus on outcomes while adjusting for multi-dimensional, dynamic, and continuous improvement in technology along the way. Teams can move quickly, respond rapidly to user needs, and produce reliable software all critical when it comes to scaling AI in the IC.

In addition, AI-related metrics must focus on key performance indicators that track the progress of how AI tools evolve rather than on only the final product to both create the opportunity for, and reflect the expectation of, value to the user earlier based on strong communication and feedback loops. Performance evaluation should center on delivery of incremental capabilities, drilling down on speed and functionality together in phases and time-boxing segmented activities, from staffing to new releases to bug-fixes.

The IC and the Committees must learn from industry best practices related to DevSecOps and software, and together develop relevant and adaptive metrics that can be consistently applied but are more aligned with AIs attributes. This joint process would itself serve as an opportunity for learning and trust building. Once developed, the metrics must continue to drive accountability and demonstrate value, and if timelines slip, the IC must quickly inform the Committees and produce new targets. The IC should expect to use the new metrics first on low-risk activities and ensure the Committees understand the standards and benchmarks the IC is using so they can evaluate programs accordingly.

While pivoting to new metrics is a good start, the IC and the Committees also must remain open to iteration, allowing oversight to change if the initial approach is less than optimal.

There is no dearth of oversight today each year, there are hundreds of written reports, in-person briefings, phone calls, hearings, and other engagements between congressional overseers and the IC. However, current oversight engagements suggest a lack of confidence and trust in the IC; they are often excessively tactical and focused on execution details that provide neither a strategic perspective on the health of a program nor an understanding of potential long-term opportunities and risks. These engagements drive a continuous cycle of meetings and briefings, requesting deeper levels of detail, in an effort to achieve the desired understanding. Unfortunately, layering detail on top of detail does not produce strategic insight, and this approach is ultimately ineffective the Committees do not feel sufficiently informed and the IC does not feel sufficiently supported, steering the relationship toward one that is more adversarial than collaborative.

Current oversight processes were not designed to be overly burdensome or act as roadblocks to progress. They were designed to give Congress appropriate insight and confidence that executive branch activities and spending are being carried out efficiently, effectively, and consistently with the law. Unfortunately, the processes have become onerous due to a history of issues that have undermined Congress trust and confidence in the IC. The IC must rebuild trust with Congress so overseers can step back from day-to-day operational details and engage with the community at a more appropriate strategic level.

The relationship between a Board of Directors (Board) and a Chief Executive Officer (CEO) in the private sector is a helpful model. The Board has ultimate responsibility for ensuring the organization is appropriately stewarding the resources entrusted to it, while the CEO manages the execution of a companys day-to-day activities. According to the KPMG Board Leadership Center, the key to a healthy relationship between a Board and the organization it oversees is trust and transparency, where the Board has constructive conversations with the leadership team about significant decisions and issues as well as the opportunity to provide meaningful input before decisions are made, and the leadership team receives valuable feedback. It is not the Boards role to see every scrap of paper that the management team sees, and it should not wade into tactical details of an issue unless the issue is related to strategy or risk.

Of course, the analogy is not perfect, but it can be instructive. The Committees, which are like the ICs Board, have the responsibility to ensure the IC is appropriately stewarding its resources. In doing so, they also have the responsibility to leverage their knowledge and expertise to provide strategic advice and direction rather than diving into many levels of detail. But, as with the private sector, without the requisite trust and transparency it is difficult for the Committees to operate at the strategic level.

To rebuild trust, the IC and the Committees must fundamentally alter the nature of their interaction, engaging not only in formal ways but also increasingly in informal engagements to better manage expectations and reduce the element of surprise. The IC should seek the Committees views on significant activities before final decisions are made and work to incorporate their feedback when possible to build stronger support and buy-in from the Committees. Formal engagements are important, but informal engagements create relationships that lead to true partnerships.

As the IC seeks additional flexibility from the Committees, it should increase trust and transparency through a more informal and open posture with Congress that includes accommodation from deliberative process privilege as needed. This would require a significant cultural shift but, if done carefully, would pay enormous dividends.

Specifically, the DNI should propose two informal, private engagements with the Committees: 1) semi-annual conversations between Committee staff and high-priority AI project leads for conversation and feedback on progress, issues, concerns, and requirements; and 2) periodic IC leadership coffee catch-ups with Committee members to better drive the strategic relationship, provide the benefit of each others thinking at that moment, and develop a sense of partnership. These engagements should not track metrics or seek to accomplish specific tasks, but rather should create mutual understanding, open dialogue, and build trust around AI activities. AI project leads should share what is known and unknown about projects, potential outcomes, and any changes in spending the Committees may see in the coming months. The IC leadership coffees would, of course, produce benefits well beyond the ICs AI activities.

It is unlikely that the information shared in these engagements would implicate the executive branchs deliberative process privilege, which reflects the presidents constitutional authority to withhold certain information from Congress before a final decision has been made within the executive branch, because these discussions would not be tied to specific decision points. Nevertheless, to help navigate these conversations the DNI should clearly set expectations with the Committees that these conversations are not formal notifications and must not be used to later reprimand the IC. The DNI should also create IC legislative engagement principles to help IC officers appropriately engage. To the extent the IC does seek pre-decisional views from the Committees, the IC should look to the accommodation process, which allows the executive branch to provide information that might otherwise be privileged if necessary to facilitate the legitimate needs of the legislative branch.

Leaning forward in this way does come with risk that the Committees will inappropriately interfere in executive branch matters. Therefore, to truly build trust, the Committees must agree to be judicious in these engagements, focus on insightful strategic and risk-based questions reflective of their extensive experience and expertise, and not misuse the information to obstruct the executive branchs authority to execute the law. Any actions to the contrary will undermine the progress made and likely end this more open dialogue. However, with agreed upon guidelines and parameters, these informal engagements would improve the AI dialogue between the IC and Congress, leading to deeper Committee understanding and, ideally, strengthening Committee support for legislation and funding of AI activities, even in times of loss or failure.

As the Committees introduce more agility into their processes and adjust their oversight to accommodate AI, they should consider the following steps to increase their confidence in the ICs activities.

First, to expand their capacity and institutional expertise, the Committees should re-organize staff along functional lines, as has already been done in some committees. Such a change would allow staff to develop a deeper understanding of various AI tools and technologies, apply that understanding strategically across IC elements, and get a more holistic cross-IC view of AI coordination and activities. While the more common model of organizing staff by IC element makes logical sense, expecting staff to understand everything an IC element does is unrealistic and unreasonable, especially given they are often single-threaded in their roles. Refocusing staff on specific functional areas and allowing them to become experts would greatly benefit not only the Committees oversight of those activities, but the IC elements they oversee. In addition, as many have recommended, Congress should recreate the Office of Technology Assessment a congressional agency that provided impartial analyses of technology and science issues to provide the Committees with access to deep technical experts when needed.

Second, the Committees should hold formal semi-annual closed substantive briefings on high-priority AI projects. In these briefings, the IC should provide enough detail for the Committees to understand progress against the new metrics, and ask questions about the strategic direction of the programs, areas of risk, concerns, unexpected issues, and future legislative and funding requirements. These briefings would provide an official mechanism for the IC to show forward movement and elevate significant issues, and for the Committees to track high-priority AI activities across the IC.

Third, if the IC receives no-year or multi-year funding for AI, the Committees should hold a focused annual review of AI spending during the previous year. This review should include an understanding of what is going well and what did not go as expected so the Committees can provide a timely and critical check on the use of that money. If the funding has been executed in accordance with congressional direction even if some of the activities have failed the money should continue to flow. If the funding has not been executed properly or consistently with congressional direction, the Committees should have the ability to stop the funding immediately.

The executive branch has significant work to do to speed and scale AI into the IC: it must reform budget and acquisition processes; create an IC AI risk assessment framework to encourage reasonable and informed risk-taking; and build an IC culture that supports innovation and accepts a level of failure. But the ICs success will be hard-fought and fleeting if the ICs congressional oversight committees do not simultaneously re-examine their supervision of the IC.

The Committees, similar to a corporate board, provide an important check on the ICs activities. To be successful in this new world of AI and emerging technology, the Committees must embrace a strategic reset, increased flexibility, and an adaptive approach to oversight. In return, the IC must lean forward with open and informal dialogue with the Committees. These adjustments will take practice to get right but, if successful, will dramatically change the ICs partnership with the Committees for the better, providing the Committees with earlier and improved insights and leading to greater support and backing for the IC.

The issues highlighted in this series are not new; countless others have raised them and good people have worked hard to solve them over many years. We cannot wait any longer for implementation to take hold. China and other adversaries are at our doorstep, and the IC must move immediately to embrace the reality of a world awash in data moving at the speed of emerging technology. Now is the time to take advantage of the groundswell of support, remove unnecessary bureaucratic barriers, and take decisive action.

* * *

Additional detail and implementation steps in all of the areas discussed in this series can be found in The Integration of Artificial Intelligence in the Intelligence Community: Necessary Steps to Scale Efforts and Speed Progress, a full-length report produced through the American University Washington College of Law Tech, Law & Security program.

Originally posted here:
Artificial Intelligence in the Intelligence Community: Oversight Must Not Be an Oversight - Just Security