Rails: Order by associate, first not downloaded

I have problem with displaying files. My app:

class User
   has_many :downloads
   has_many :downloaded_files, :through => :downloads, source: :file
end

class File 
   attr_accessor :downloaded

   has_many :downloads
   has_many :users, :through => :downloads
end

class Download
   belongs_to :user
   belongs_to :file
end

And when user log in i want to display all files but downloaded files by current user should be the last. The file table is about 500 records so i need simple and fast method. Have someone any idea ?

Rails 4.2.1 Ruby 2.2.2 Postgresql