Exploring Monster Taming Mechanics in Final Fantasy XIII-2: Relational Data

In this next installment of the miniseries of exploring the monster taming mechanics of Final Fantasy XIII-2, we'll fill out another database table that we need in order to start connecting all of the monster data together. In the last article, we built the core monster table with hundreds of attributes for each of 164 monsters. In the first article, we had identified four other tables that we would need as well, these being abilities, game areas, monster materials, and monster characteristics. The data in these four tables is all related in one way or another to the monsters in the monster table. We'll start with the abilities table, which will end up being three tables because we actually have passive, command, and role abilities. Once the passive abilities table is complete, we'll see how to connect that data in the database so that we can later make inferences on the data.

Finding Monster Abilities

Like for the core monster data, the first thing we need to do is get the data for the monster abilities into a .csv format. We'll start with the passive abilities because this is the largest group of abilities and they have a rank attribute that the role and hidden abilities don't have. Since there are over 200 passive abilities, we're going to want to pull this data out of the FAQ with another script instead of building it up manually. We can use an FSM very similar to the one used for parsing the monster data with a few simplifications because there are not nearly so many attributes for the passive abilities. I'll just throw the script out there, and then we can discuss it:

SECTION_TAG = "PassADe"
ABILITY_SEPARATOR = "........................................"
ABILITY_REGEX = /(\w\S+(?:\s[^\s\(]+)*)\s\((RL|\d)\)-*:\s(\w\S+(?:\s\S+)*)/
ABILITY_EXT_REGEX = /^\s+(\S+(?:\s\S+)*)/

end_abilities = lambda do |line, data|
  return end_abilities, data
end

new_ability = lambda do |line, data|
  props = line.scan(ABILITY_REGEX)
  if props.empty?
    if line.include? ABILITY_SEPARATOR
      return end_abilities, data
    else
      extra_line = ABILITY_EXT_REGEX.match(line)
      data.last["description"] += ' ' + extra_line[1]
      return new_ability, data
    end
  end

  if props.first[1] == "RL"
    props.first[1] = "99"
    props.first[0] += " (RL)"
  end
  data << {"name" => props.first[0], "rank" => props.first[1], "description" => props.first[2]}
  return new_ability, data
end

find_abilities = lambda do |line, data|
  if line.include? ABILITY_SEPARATOR
    return new_ability, data
  end
  return find_abilities, data
end

find_sub_section = lambda do |line, data|
  if line.include? SECTION_TAG
    return find_abilities, data
  end
  return find_sub_section, data
end

section_tag_found = lambda do |line, data|
  if line.include? SECTION_TAG
    return find_sub_section, data
  end
  return section_tag_found, data
end

start = lambda do |line, data|
  if line.include? SECTION_TAG
    return section_tag_found, data
  end
  return start, data
end

next_state = start
data = []
File.foreach("ffiii2_monster_taming_faq.txt") do |line|
  next_state, data = next_state.(line, data)
end

Starting from the bottom, we can see that we kickoff the FSM by looking for the SECTION_TAG, which is "PassADe" in this case. We have to find it three times instead of the two times we looked for the section tag in the monster FSM because the tag appears one extra time in the FAQ in a sub-contents under the main "Monster Infusion" section. Then, we look for the ABILITY_SEPARATOR, which is the same as the MONSTER_SEPARATOR from the previous script, and then go into a loop of matching and storing the ability data in the data list of hash tables. The ability data has a pretty clean format with the possibility of overflowing to a second line. The first couple abilities showcase the variations we have to handle:

Uncapped Damage (RL)---------: Raises the cap on damage to 999,999.
Enhanced Commando (RL)-------: Enhances the Commando Role Bonus by one Bonus
                               Boost.

We have a name that can be multiple words (and by looking further ahead we also see some special characters) followed by a space and a rank in parentheses, a -: separator, and the description, possibly extending to a second line. We want to capture the name, rank, and description, and the rank can be either "RL" (for Red Lock, meaning it can never be removed from the monster) or a number 1-9. From what we've learned of regexes in past posts, it should be clear that the regex

/(\w\S+(?:\s[^\s\(]+)*)\s\((RL|\d)\)-*:\s(\w\S+(?:\s\S+)*)/

is sufficient to match on the first line and capture those three attributes. Looking at the rest of the new_ability state code, we can see that if the line doesn't match the ABILITY_REGEX, it checks if the line is the ABILITY_SEPARATOR and moves to the last state if it is. Otherwise, we know that the line is an extra line of description, so we capture it and add it to the description of the last ability captured. If the line did match the ABILITY_REGEX, then we create a new hash with the values populated from the regex captures and continue in the same state.

Note that if the rank was "RL", we convert that to "99" so that all ranks will be integers in the database. Why not "10?" Well, when an ability is yellow-locked, its rank increases by nine, so making the red-locked abilities have a rank of 10 would conflict. I could have made it 19 or 20, but 99 works just as well and really sets them apart. We also need to add the " (RL)" text back into the name of the ability because it's possible to have red-locked and non-red-locked abilities with the same name, and they are different abilities for all intents and purposes.

Validating and Exporting Monter Abilities

Next, we want to do a little validation check on this data and write it to a .csv file, so here's the code to do that:

PROPER_NAME_REGEX = /^\w.*[\w)!%]$/
NUMBER_REGEX = /^\d\d?$/
FREE_TEXT_REGEX = /^\S+(?:\s\S+)*$/

VALID_ABILITY = {
  "name" => PROPER_NAME_REGEX,
  "rank" => NUMBER_REGEX,
  "description" => FREE_TEXT_REGEX,
}

data.each do |ability|
  VALID_ABILITY.each do |key, regex|
    if ability.key?(key)
      unless ability[key] =~ regex
        puts "Monster ability #{ability["name"]} has invalid property #{key}: #{ability[key]}."
      end
    else
      puts "Monster ability #{ability["name"]} has missing property #{key}."
    end
  end
end

require 'csv'
opts = {headers: data.first.keys, write_headers: true}
CSV.open("monster_abilities.csv", "wb", opts) do |csv|
  data.each { |hash| csv << hash }
end

Since we're building up each hash table directly with name, rank, and description, it's not necessary to make sure that each hash contains all three attributes and no others. We just want to make sure that each attribute value has the right (admittedly loose) format. Then we write it out to the monster_abilities.csv file.

Importing Monster Abilities

Moving right along, we can now import this .csv file into the database with another simple addition to our seed script:

csv_file_path = 'db/monster_abilities.csv'

CSV.foreach(csv_file_path, {headers: true}) do |row|
  Ability.create!(row.to_hash)
  puts "#{row['name']} added!"
end

Before running this script, we need to generate the Ability model and its schema. Since there are only three attributes, we can easily create the migration script in one shot:

$ rails generate model Ability name:string rank:integer description:string

This command will create this migration file:

class CreateAbilities < ActiveRecord::Migration[6.0]
  def change
    create_table :abilities do |t|
      t.string :name
      t.integer :rank
      t.string :description

      t.timestamps
    end
  end
end

Now we can run the migration and then seed the database:

$ rails db:migrate
$ rails db:seed

And we've added our second table to the database with 147 abilities. (Wait, I thought there were over 200 abilities. Good catch. We'll get to that in a bit.) However, we should have a link between this new ability table and the monster table because every monster's many abilities should all be one of the abilities in the ability table.

Associating Monsters' Abilities with the Abilities Table

What we want to do in order to associate each monster's abilities with the abilities table is make each ability in the monster table a reference instead of a string, and point those references at the correct specific abilities in the abilities table. To accomplish this association, we'll need to change both the monster table migration and the monster model part of the seed script.

First, we can change all of the t.string declarations in the monster table migration to t.references. Changing only the abilities that end in "_passive" is sufficient for now because those are the abilities we created in the abilities table. The references for these abilities will actually be foreign keys from the abilities table, so we need to tell Rails that at the end of the migration:

class CreateMonsters < ActiveRecord::Migration[6.0]
  def change
    create_table :monsters do |t|
      t.string :name
      # ... The other monster attributes ...
      t.references :default_passive1
      t.references :default_passive2
      t.references :default_passive3
      t.references :default_passive4
      t.string :default_skill
      t.string :special_notes
      t.references :lv_02_passive
      t.string :lv_02_skill
      # ... The rest of the lv XX abilities ...

      t.timestamps
    end

    add_foreign_key :monsters, :abilities, column: :default_passive1_id, primary_key: :id
    add_foreign_key :monsters, :abilities, column: :default_passive2_id, primary_key: :id
    add_foreign_key :monsters, :abilities, column: :default_passive3_id, primary_key: :id
    add_foreign_key :monsters, :abilities, column: :default_passive4_id, primary_key: :id
    add_foreign_key :monsters, :abilities, column: :lv_02_passive_id, primary_key: :id
    # ... All of the other _passive abilities foreign keys ...
  end
end

I know this seems like tedious busy work, but it's necessary for setting up the database. Some of it can be done with regex replace tools in some editors or generated with a script. I just brute-forced it and got pretty good at changing numbers really rapidly. Be sure to remove lv_xx_passive abilities that don't exist later in the list.

Because we changed this migration, we need to rerun it, but now it references the ability migration so when we rollback the migrations, we need to change the date in the file name on the monster migration to be after the ability migration before we run the migrations forward again. Otherwise, Rails will complain.

$ rails db:rollback STEP=2
$ [rename the monster migration file to a date after the ability migration file]
$ rails db:migrate

We're not quite done with preparing the database, yet, because the Rails models need to know about this association, too. The app/models/monster.rb file needs to know that the passive ability attributes are associated with the Ability model using the belongs_to association:

class Monster < ApplicationRecord
  belongs_to :default_passive1, class_name: 'Ability', optional: true
  belongs_to :default_passive2, class_name: 'Ability', optional: true
  belongs_to :default_passive3, class_name: 'Ability', optional: true
  belongs_to :default_passive4, class_name: 'Ability', optional: true
  belongs_to :lv_02_passive, class_name: 'Ability', optional: true
  # ... The rest of the lv_XX_passive attributes ...
end

It may sound weird that a monster belongs to an ability, but that's the type of association we want where the link points from the monster attribute to the ability. It's the same as if we had an Author model and a Book model, and the book belongs to the author that wrote it. The link would be in the book with the author's ID. That same link direction is what we want with monsters having links to ability IDs, so belongs_to it is.

We also need to add associations to the Ability model so that we can follow links from abilities to monsters. That association is done with has_many, which also seems a bit weird, but oh well:

class Ability < ApplicationRecord
  has_many :default_passive1_monsters, :class_name => 'Monster', :foreign_key => 'default_passive1'
  has_many :default_passive2_monsters, :class_name => 'Monster', :foreign_key => 'default_passive2'
  has_many :default_passive3_monsters, :class_name => 'Monster', :foreign_key => 'default_passive3'
  has_many :default_passive4_monsters, :class_name => 'Monster', :foreign_key => 'default_passive4'
  has_many :lv_02_passive_monsters, :class_name => 'Monster', :foreign_key => 'lv_02_passive'
  # ... The rest of the lv_XX_passive attributes ...
end

That was the easy, if mindlessly tedious, part. The next step is trickier. We want to change the db seed script so that when the monsters are created, the passive abilities in the monster table are references to the abilities table, and we want to catch any instances where the ability doesn't exist, meaning there was a typo or an omission. We need to make sure the ability table is populated first in the script, so we have access to those abilities when we're importing the monsters. Then, for each passive ability for each monster we search the ability table for that ability's name, and assign it to the corresponding ability attribute for the monster. That assignment creates the proper reference. If the ability isn't found, we print an error and return so the error can be fixed in the FAQ text file. We'll then have to rerun the ability or monster parser and try the import again. Here's what this process looks like in code, including the ability import:

csv_file_path = 'db/monster_abilities.csv'

CSV.foreach(csv_file_path, {headers: true}) do |row|
  Ability.create!(row.to_hash)
  puts "#{row['name']} added!"
end

csv_file_path = 'db/monsters.csv'

CSV.foreach(csv_file_path, {headers: true}) do |row|
  monster = row.to_hash
  monster.keys.select { |key| key.ends_with? '_passive' }.each do |key|
    if monster[key]
      monster[key] = Ability.find_by(name: monster[key])
      if monster[key].nil?
        puts "ERROR: monster #{monster['name']} #{key} not found!"
        return
      end
      puts "Found #{key} #{monster[key].name}"
    end
  end
  Monster.create!(monster)
  puts "#{row['name']} added!"
end

This code is pretty much written as just described, but note that we check if the monster[key] exists before doing anything with it. Remember that these are optional attributes, so they can be nil for any given monster ability. If we run this script, we'll find right away that Cactuaroni has an ability "Critical: Haste (RL)" that doesn't exist in the ability table. We could add this ability to the FAQ and rerun the parser, but notice that there is an ability "Critical: Haste" already in the table. This issue of the non-red-locked ability existing in the table but not the red-locked version is fairly common, so we could solve the problem with code instead of tediously rerunning the scripts to find all of the instances where the red-locked ability is missing. All we have to do is search for the base ability, copy it, change the rank to 99, and tack " (RL)" onto the name. This process can be done like so:

CSV.foreach(csv_file_path, {headers: true}) do |row|
  monster = row.to_hash
  monster.keys.select { |key| key.include? '_passive' }.each do |key|
    if monster[key]
      if monster[key].ends_with?(' (RL)') && Ability.find_by(name: monster[key]).nil?
        ability_name = monster[key][0..-6]
        puts "Searching for #{key} ability #{ability_name}"
        ability = Ability.find_by(name: ability_name).dup
        ability['name'] = monster[key]
        ability['rank'] = '99'
        ability.save
        puts "Ability #{ability['name']} added!"
      end

      monster[key] = Ability.find_by(name: monster[key])
      if monster[key].nil?
        puts "ERROR: monster #{monster['name']} #{key} not found!"
        return
      end
      puts "Found #{key} #{monster[key].name}"
    end
  end
  Monster.create!(monster)
  puts "#{row['name']} added!"
end

If the monster ability ends with " (RL)" and it's not in the ability table, then we copy the base ability and create the corresponding red-locked ability from it. We've sidestepped a bunch of dreary work with a little bit of targeted code!

Now after running this script to seed the database, we will find a few more errors in the FAQ to fix. Four red-locked abilities are missing, so we'll have to add Perpetual Poison, Resist Damage +05%, Bonus CP, and Feral Speed. I found definitions for these abilities simply by googling them. The Resist Elements +20% ability that Twilight Odin has was also missing. I guessed at the rank of 8 for this one because I couldn't find it. Resist Elements +30% has a rank of 9 and Resist Elements +05% has a rank of 5, so it's most likely rank 7 or 8. Finally, there were three typos: the ability "Auto: Enfire (RL)" should not have the colon, the ability "Auto Haste (RL)" should be hyphenated, and the ability "ATB: Advantage (RL)" should not have the colon. After everything is fixed, we have a complete ability table with 218 abilities, and every passive monster ability is linked correctly to the ability table.


That was quite a lot of work, some of it tedious, but we accomplished a lot. We generated a list of passive abilities from the FAQ, validated that data, imported it into a new table in the database, and linked all of those abilities to the monsters that can learn them in the monster table. This process can be repeated for the much smaller tables that are left to create, and that is what we'll do next time.

Exploring Monster Taming Mechanics Table of Contents:

Part 1: Introduction

Part 2: Data Collection

Part 3: Data Validation and Database Import

Part 4: Relational Data

Part 5: The Remaining Tables

Part 6: Viewing Data

Part 7: Viewing More Data

Part 8: Viewing the Monster Data

Part 9: Viewing More Monster Data and Abilities

Part 10: Filtering Monsters

Part 11: Asking Deeper Questions

Part 12: Finding Monster Materials

Part 13: Monster Characteristics

Part 14: Dogfooding