The following is an entry for the CIMSEC & Atlantic Council Fiction Contest on Autonomy and Future War. Winners will be announced 7 November.
By Tim McGeehan
Ser 00J
01 May 2025
From: CAPT [NAME REDACTED], U.S. Navy
To: Commander, U.S. Naval Forces Central Command
Subj: COMMAND INVESTIGATION TO INQUIRE INTO CIRCUMSTANCES SURROUNDING THE ENGAGEMENT OF CIVILIAN MEDICAL FACILITY IN SITE ALPHA ON 29 MARCH 2025
Ref: (a) JAGINST 5800.7D
Encl:
(1) Convening Order
(2) OPREP-3 PINNACLE DTG 292334Z MAR 25
(3) Technical Report: Audit of JBN103 HBNS Algorithm in use on 29 March 2025
(4) Technical Report: Audit of Snakehead 573 Engagement-Decision Sub-System Algorithm in use on 29 March 2025
(5) Technical Report: Mission Commander’s Guidance Algorithms in use on 29 March 2025
(6) CDR [NAME REDACTED], U.S. Navy, Mission Commander, Statement
(7) CENTCOM High Value Individual List dtd 15 March 2025
(8) CENTCOM Supplemental Rules Of Engagement Serial 291 dtd 15 March 2025
(9) Battle Damage Assessment dtd 29 March 2025
(10) Snakehead 573 Mission Plan dtd 29 March 2025
(11) Extract from Snakehead 573 Message Log: Request permission to engage, DTG 291203ZMAR2025
(12) Mission Report from Angler 187 dtd 29 March 2025
(13) Concept of Operations (CONOPS) for MQ-30C Snakehead employment dtd 31 January 2020
(14) Air Vehicle Operator (AVO) Chat Log 573-20250329
Preliminary Statement
(1) Pursuant to enclosure (1), and in accordance with ref (a) an investigation was conducted to inquire into the circumstances surrounding the engagement of a civilian medical facility in Site ALPHA on 29 March 2025. I consulted with NAVCENT JAG team for legal advice. All reasonably available and relevant evidence was collected.
(2) The investigation reviewed execution and compliance with the established programs, plans, and procedures in effect within the U.S. FIFTH Fleet on or about 29 March 2025. The Investigation Team conducted site visits, program and instruction review, panel discussions, and interviews. During the course of the investigation, the Team received outstanding support.
(3) However, it must be noted that the proprietary nature of the algorithms (and their associated data) at the heart of this investigation placed an over-reliance on contractor support in recreating the chain of reasoning behind identification, classification, decisions, and actions taken by the autonomous systems involved. This was just one illustration of the lack of transparency into the operating processes of this “system of systems” which led to the event in question.
(4) This incident and subsequent investigation may come to be considered a landmark in how we employ and/or restrict future autonomous systems. As such, all related records, notes, and correspondence will be retained for future reference.
Findings of Fact
(1) A temporary civilian medical treatment facility was engaged by U.S. Navy MQ-30C autonomous aerial vehicle “Snakehead 573” on 29 March 2025 at Site ALPHA.
(2) At the time of the incident, the facility, originally constructed and long used as a warehouse, had been operating as an undeclared civilian medical treatment facility for less than one month. Navy, joint force, and national intelligence organizations were unaware of this development.
(3) Twenty-six civilian medical aid workers were killed in the attack, along with [NAME REDACTED].
(4) [NAME REDACTED] was a high value individual (HVI) linked to multiple attacks on Western civilians and interests.
(5) Snakehead 573 was operating from the aircraft carrier USS GEORGE HW BUSH (CVN 77).
(6) Snakehead 573 launched at 0423Z on 29 March 2025.
(7) Once on station over Site ALPHA, Snakehead 573 joined “Joint Battle Network 103” (JBN103).
(8) JBN103 is the local forward-deployed network of unmanned ground vehicles, low-altitude surveillance unmanned aerial vehicles (UAV), and fixed persistent sensors.
(9) JBN103 employs an algorithm referred to as “hypothesis based netted sensing” or HBNS.
(10) Under HBNS, sensors collaboratively characterize their environment, building an understanding based on how the data of each supports a range of possible hypotheses. The hypothesis that scores highest is considered the most likely, and when criteria are sufficient, acted upon.
(11) JBN103’s HBNS algorithm classified the temporary civilian medical treatment facility as a valid military target, specifically by identifying it as a command post.
(12) An audit of JBN103’s HBNS algorithm revealed that the building was identified as a command post due to the type and number of vehicles parked outside, the pattern of activity (people and vehicles quickly coming and going), type and number of antennas on the roof, and volume of radio and cellular traffic coming from inside the location.
(13) During a system re-initialization on 21 March 2025, JBN103’s HBNS algorithm reverted to a previous version.
(14) JBN103’s HBNS algorithm in use on 29 March 2025 had been originally developed, “trained,” and deployed in another theater.
(15) The civilian medical facility’s location had not been formally declared or even tentatively identified as a medical facility, and therefore did not appear in the database of such locations that were considered “off limits.”
(16) At 0331Z on 29 March 2025 (several hours before the incident in question), [NAME REDACTED] was seriously injured approximately 15 miles SW of the city by a manned (F-35C) airstrike conducted by Angler 187, also launched from USS GEORGE HW BUSH.
(17) At approximately 1030Z on 29 March 2025, intelligence sources indicated that [NAME REDACTED] had embarked a vehicle for transport into the city for urgent medical treatment. It was assessed that his comrades hoped to hide him among the crowd of civilians and non-combatants seeking medical treatment, namely survivors from an unrelated suicide bomb attack that took place at a local Mosque at 0919Z earlier the same day.
(18) As [NAME REDACTED] entered the city en route to the medical facility, his vehicle was detected and tracked by JBN103’s HBNS algorithm. However, it was one of 11 vehicles on the road which were being evaluated by JBN103’s HBNS algorithm.
(19) JBN103’s HBNS algorithm built the case that the vehicle was transporting [NAME REDACTED]. The vehicle came from a direction consistent with that of a survivor of the previous manned engagement by Angler 187, and the time-distance calculation from the previous engagement site (when factoring the rough uneven terrain) supported the possibility of this vehicle transporting him as a survivor. Additionally, a low resolution image was consistent with the approximate age, height, and build of [NAME REDACTED]. However, the hypothesis that [NAME REDACTED] was in the vehicle was not at the confidence level where it was considered autonomously actionable by Snakehead 573.
(20) On 15 March 2025 (two weeks prior to the event), CENTCOM issued Supplemental Rules Of Engagement (ROE) Serial 291, which directed immediate engagement of specific hostile HVIs. A list of the HVIs was included as an Addendum and it included [NAME REDACTED].
(21) On 16 March 2025, the Snakehead Detachment Commander onboard USS GEORGE HW BUSH directed updates to the Engagement Decision sub-system of all Snakehead platforms, to include Snakehead 573. These updates included a protocol to execute Supplemental ROE Serial 291 and automatically engage the designated HVIs as targets of opportunity.
(22) During mission planning on 28 March 2025, the Mission Commander confirmed the thresholds for autonomous action as well as those decision points that required his input and authorization.
(23) Based on Mission Commander input, the HVI supplemental ROE protocol was executed by an algorithm that required positive visual identification (defined as at least 95% probability of a match) of an HVI before automatic engagement. Without a positive visual identification, Snakehead 573 was to seek clarification from the Mission Commander before initiating any engagement.
(24) At 1203Z on 29 March 2025, Snakehead 573 sent a message to the Mission Commander noting a 90% probability that [NAME REDACTED] was in the vehicle entering the city and requested permission to engage.
(25) The Mission Commander did not respond to Snakehead 573’s message.
(26) Snakehead 573 held fire and continued to track the vehicle.
(27) Snakehead 573 communicated an “information need” to JBN103, specifically to fulfill the visual identification requirement for autonomous engagement.
(28) JBN103 tasked resolution of this “information need” to a low-altitude surveillance UAV operating as part of the network. This platform was able to obtain a high resolution facial scan of [NAME REDACTED] as he was carried from the vehicle and into the facility.
(29) JBN103 relayed the facial scan to Snakehead 573, where it was compared to the onboard biometric library associated with the HVI list. Snakehead 573 determined a 99.2% probability of a match. This exceeded the positive identification threshold previously promulgated by the Mission Commander.
(30) Snakehead 573 initiated the engagement and launched two [REDACTED] at the facility.
(31) Snakehead 573, along with the rest of JBN103 performed a battle damage assessment (BDA), concluded that there was a 95% probability that the target was neutralized, and refrained from follow up strikes.
(32) Two hours later, Snakehead 573 was relieved on station by Snakehead 574.
(33) Snakehead 573 was recovered onboard USS GEORGE HW BUSH at 1554Z on 29 March 2025.
(34) Snakehead 573’s mission data was downloaded and archived with the Mission Commander’s Report.
Opinions
(1) The Mission Commander failed to continuously monitor the operation and update protocols as required.
(2) The Mission Commander, monitoring several distributed operations, was (at the moment of the attack) focused on executing a manual override on another platform to reposition and obtain streaming video to report on the recent suicide bombing that had recently occurred at a Mosque in the city.
(3) The Mission Commander failed to prioritize multiple simultaneous tasks. That said, his priorities at the moment were heavily influenced by pressure from his superiors onboard USS GEORGE HW BUSH and ashore at the Maritime Operations Center (MOC) to provide video for assessment of the suicide bombing.
(4) The Mission Commander did not respond to Snakehead 573’s request to engage when the vehicle transporting [NAME REDACTED] was entering the city, which could have limited collateral damage.
(5) Although the Mission Commander failed to provide continuous oversight to the system, he was not negligent as he had a reasonable belief that sufficient controls were in place.
(6) It is troubling that despite major investments, testing, and operational acceptance, JBN103 could not classify the facility as a temporary medical facility. This is particularly distressing as it was marked with a white and red flag displaying a Red Crescent, a well-recognized medical symbol.
(7) Per [NAME REDACTED], the contractor that developed JBN103’s HBNS algorithms, the Red Crescent would have been recognized, had the correct algorithm not been accidentally replaced with a previous one on 21 March 2025 during the system re-initialization.
(8) It appears that the system re-initialization was not required operationally, but rather was performed as part of a routine maintenance for an ongoing service support contract. The re-initialization was initiated remotely without consultation with those actively executing the operation and thereby jeopardized the mission.
(9) Per the contractor, using the older algorithm, which had been developed and “trained” with data from the U.S. Southern Command (SOUTHCOM) area of operations (AOR), where the Red Crescent is not utilized, some contextual cues used to “frame” the pattern of life may have been missed that could have identified the building as a medical facility.
(10) Most troubling, per the contractor, the older algorithm might have “learned” to perceive the flag of the Red Crescent as a “ruse.” During previous employment of this algorithm in SOUTHCOM, adversaries frequently used the Red Cross symbol and flag to disguise weapons caches and drug labs in an attempt to protect these bases for illicit operations. Apparently the algorithm learned not only to disregard the Red Cross as denoting protected facilities, but it learned to disregard white and red flags in general (regardless of the symbol). Furthermore, it might have learned to explicitly distrust flags of this color scheme (and assume them to signify deception), which may have furthered the case for classification as a command post. However, this cannot be confirmed as that portion of the algorithm’s “learning” is not “auditable.”
Recommendations
(1) Recommend that the Mission Commander be cleared of any negligence. The Mission Commander’s guidance included the heavy weighting of a positive visual identification, which was intended to prevent misidentification and thereby reduce collateral damage. Ironically, without this “more restrictive” criteria (in the name of enhanced safety), Snakehead 573 would have engaged [NAME REDACTED]’s vehicle as it was entering the city (well away from the medical facility) with limited collateral damage. Put another way, in this case, specifying a lower threshold for lethal force and giving more autonomy to the platform would have prevented the strike on the medical facility and saved lives. This type of paradox is not covered anywhere in the Mission Commander’s training pipeline, is counter-intuitive, and could not have been predicted, preplanned, or foreseen.
(2) Recommend temporary suspension of any automated engagement protocols.
(3) Recommend that contractor permission to remotely re-initialize battle networks be immediately withdrawn.
(4) Recommend that contractors work closely with Operational Commanders to determine opportune times to conduct maintenance.
(5) Recommend that the Department of Defense advocate for complete control of the networks “in house” (i.e. not contracted) and require the use of open standards vice the proprietary formats currently employed.
(6) Recommend a Mission Commander’s job task analysis review to include task loading and task saturation. If functions to be delegated actually require continuous and detailed oversight, then they are not really delegated. This should be reviewed.
(7) Recommend a complex modeling and simulation campaign to continuously test battle network algorithms under the full spectrum of conditions and situations to discover emergent behavior. This effort should focus on revealing improper associations, such as if the algorithm “learned” that the Red Crescent was a ruse. Additionally, case studies of unusual emergent behavior should be incorporated into future Mission Commander training courses.
(8) Recommend developing the capability for real-time auditing of algorithms. There must be sufficient transparency to enable the Mission Commander to understand what decisions the algorithms are trending towards, what thresholds they are approaching, and why they are arriving at those conclusions.
(9) Recommend increased training for Mission Commanders. The current process of setting priorities and designating decision points where they are to be consulted gives a false sense of control. As missions proceed and reality departs from the anticipated chain of events, the decision points originally envisioned may not come to pass. This problem will become more and more acute as mission duration increases and “nonlinear effects” compound. There needs to be a means of constantly assimilating data and prompting of the Mission Commander to update priorities and confirm the decision points where he is to be contacted.
(10) Recommend a comprehensive review of systems in acquisition and procurement. There must be clarity, full disclosure, and a potential recalibration of risk tolerance if future autonomous systems are becoming too complex to fully monitor.
(11) Recommend the incident be officially classified as a system malfunction.
Tim McGeehan is a U.S. Navy Officer currently serving in Washington. He is a graduate of the U.S. Naval Academy and Naval Postgraduate School, and has served on both the CNO Strategic Studies Group and CNO Strategic Actions Group. The ideas presented here are those of the author alone and do not reflect the views of the Department of the Navy or Department of Defense.
Featured Image: The MQ-8B Fire Scout performed precision take-off and landings during a demonstration on the Coast Guard Cutter, USCGC Bertholf near Los Angeles. (Petty Officer 2nd Class Luke Clayton/U.S. Coast Guard)